VMWorld Highlight: Virtual Infrastructure Optimization

The Solutions Exchange at VMWorld Europe confirms that many vendors are tapping into a key customer concern for 2009: the optimization of existing, growing virtual server estates and the integration of storage performance data into the administrator's dashboard. While many customers I've spoken to have made progress toward managing virtual machine sprawl, they struggle to identify, correlate, and diagnose performance problems for I/O-intensive production applications, problems that often span server to storage.

From the application through the I/O adapters and switches, to the arrays themselves, there's a lack of visibility into the I/O path for root cause analysis -- in real-time and at production scale. This is in addition to the problem of right-sizing: resource optimization at provisioning/set-up times. Both activities require deeper insight into the impact of virtualization on storage performance.

Virtual Instruments' VirtualWisdom, announced here this week, aims to provide this end-to-end runtime visibility and cut through the finger-pointing between server and storage vendors when performance issues arise in production. Their solution is worth a look, to augment in-place or other vendor solutions (there are plenty of excellent ones here to explore) that may provide insight only during the initial capacity planning and provisioning phases. It's clear that even the best-planned virtual environment often behaves differently at production scale; VirtualWisdom can tell you why.

Backup and Archive: Two Different Animals

Are you using older backups as your archives? Are your archives sitting on tape? For years, this has been the norm because on the surface this approach looks cheap and easy. But like some other things that are cheap and easy, you may be in for a few unwanted surprises if you continue in your errant ways.

Historically, backup and archiving were mostly about making and retaining a copy of the production data. Many backup products in the 90s seemed to be all about backup, regardless of what that did to the recovery process. Archives, if people even had them, were about keeping data around as cheaply as possible, mostly to meet regulatory requirements (which meant that it was mostly done in certain industries with compliance requirements). Both used tape, so it was natural for a "backup" to become an "archive" and get shipped to some remote site after some period of time. But things have changed considerably:

1) With increasingly stringent requirements for RPO and RTO, the focus of data protection has clearly shifted to recovery
2) Operations have moved to a 7x24 clock, driving concerns about the implications of backup on production application environments
3) The focus of archiving has expanded to include accessibility, primarily to meet the demands of an increasingly litigious corporate environment

Pushing the envelope on tape technologies to try to address the first two items above led to another unintended consequence: people became very aware of the recovery reliability issues with tape media when used to meet backup requirements. Tape is a sequential access media, but backups and restores basically needed a random access media. Tape is also primarily an offline medium, a fact which meant it did not lend itself well to the types of discovery operations that had to be performed against archives to find responsive materials to deal with lawsuits. A study we did last year indicated that discovery operations against tape cost 10x as much as those same operations if they were performed against disk where computerized search could be leveraged. With the average cost of a lawsuit being in the range of half a million dollars for large enterprises, e-discovery could save hundreds of thousands of dollars if at least several lawsuits were being handled per year. Plus, imagine the judge's reaction when you can't produce some responsive materials that you clearly should be able to due to media reliability issues. Disk was the obvious answer, if its cost could be brought down significantly.

Today, backup is about recovery, archiving is about cost effective retention and searchability. The two business objectives drive different requirements, but there is a single medium which is well matched with their foundation requirements: disk. Different software functionality is required for each, but this raises the question again of whether your backups should just age into becoming your archives.

We recommend that backup and archive be managed separately. First, since most restore requests come from the most recent backups, the "backup" problem has more of a short term focus to it. Disaster recovery has less of a short term focus, mostly because of operational limitations about how to get that data to a remote site but also because of the requirement that it support multiple comprehensive recovery points. Archiving clearly has a long term focus but should NOT just be a process which occurs at the end of the backup data life cycle. To optimize your existing storage infrastructure for performance, cost, and protection, data should be archived well before it is no longer needed for backup and/or DR purposes. This drives very positive implications for managing primary storage and the costs associated with it (see my blog from February 24, 2009).

In dealing with end users on this issue, two conclusions are evident:

* Backups and archives should be managed separately, and you should seriously consider using disk-based options for both if you're not already
* Archiving to tape is NOT cost effective from an overall TCO point of view if you're dealing with multiple concurrent lawsuits on a regular basis

Pulling One Out Of The Hat

Inertia is a fact of corporate life. If you're responsible for figuring out what you're going to spend your IT budget on in the next 12 to 18 months, it's likely that a very high percentage of your spend will be on projects that you were spending money on last year. When trying to find places to cut, many people think first about new projects and initiatives. But another great place to look are budgets that get a large relative percentage of your overall spend. And if you're like most enterprises in this era of exploding data growth and increasing retention requirements, your primary storage budget probably fits that definition.

If the corporate stars have aligned and everyone is already in agreement that you're going to spend "x" on primary storage this year, here's something to consider. The business objective is to meet the enterprise's requirements for primary storage capacity, along with the infrastructure requirements that go along with that (data protection, DR, security, etc.). As long as you meet that requirement within the budget, you may have some flexibility in whether you spend that on technology that meets the strict definition of "primary storage" as long as you meet the business requirement. It's a fact that for most enterprises, at least 70% of the data sitting in primary storage infrastructure today is rarely if ever accessed. But it's there on the off-chance that you might need it, and it hasn't gotten to the point that you're so sure you'll never need it that you've migrated it to some sort of tape archive. All that data is driving a lot of management overhead for you for performance optimization, redundancy, backup, etc.

More and more vendors have figured this out, and are offering what we at Taneja Group call "active archival storage". Basically, these are very scalable, disk-based platforms, generally accessible through industry standard interfaces such as NFS and CIFS, that leverage technologies like SATA and storage capacity optimization (file level single instancing, data de-duplication, compression, etc.) to store lots of data very cheaply. Permabit might have been one of the first to enter this game, but they've been joined by other vendors, small and large, and you can now deploy this capability either as a product or as a service (Iron Mountain recently announced a cloud-based archiving solution). Here's the thinking: you've already decided you need x TB of new primary storage this year (fill in your requirement) and that's going to cost y dollars (fill in your cost). Instead of buying more primary storage, take that y dollars and buy an active archiving platform (or start up one of the cloud-based services) and move the 70% or more of your stale "primary" data into it. That has several impacts:

1) now you need a lot less primary storage (and probably won't need to buy any more this year after you've freed up 70% of your existing primary storage capacity)
2) now you're backing up a lot less primary storage, so backups take a lot less time
3) now you're spending a lot less to store your data, since the new average $/GB is a blend between the $20/GB or more you're paying for primary and the $1/GB or less you'll be paying for this active archiving platform (think how much more active archive storage that $20/GB will buy)
4) the data is still online so end users can transparently access it, and because it's online it's now searchable for e-discovery purposes, a fact which we've seen save hundreds of thousands of dollars (relative to tape-based discovery) in just one year for large enterprises that deal with multiple lawsuits concurrently (which unfortunately is most of the Fortune 2000)

It's not all peaches and cream, though, since you'll have to manage another platform, which may or may not mean introducing a new vendor into your shop. But you can do this without asking for any additional budget and you'll be easing the backup burden while at the same time decreasing e-discovery costs in a big way, not to mention making it faster and easier. Data migration doesn't need to be done up front, you can just let the platform manage that over time according to policies you establish. If you're going to buy one of these platforms, though, you'll need sufficient scale, say around 80-100TB of primary storage with your data growing at a good clip, to cost justify it using the above example. Smaller companies may consider cloud-based offerings which will let you in for under 1TB.

Long term, most medium to large enterprises will be using an online secondary storage tier. Tape just can't meet evolving archive requirements, especially where e-discovery is a concern. With the economy the way it is these days, this is something to at least think about this year.

VMworld Europe 2009: Keynote Live Blogging

VMware CEO Paul Maritz kicked off VMworld Europe today and fired a shot directly across Citrix's bow. You may recall Citrix's Jan 21 Announcement of a bare-metal desktop hypervisor to be developed in partnership with Intel. Now, just a little over a month later, VMware signals it's not ready to give up the desktop by making virtually the same announcement. 

VMware's Client Virtualization Technology, leveraging Intel's vPro, is a direct match for Citrix's Project Independence. Maritz was clear that the desktop is key to VMware's 2009 strategy, adding that the View (VDI) suite announced in December will be fully rolled out by the end of 2009.

Also, Maritz premiered VMware vSphere, a blanket rebranding of the VI suite which seems to include (replace?) the Virtual Data Center Operating System (VDC-OS) branding of 2008. At first glance, the new brand aims to break down any distinction between internal, external and "private" clouds: they are all one extended virtual fabric infrastructure. This led to a sweeping vision of VMware's vCenter management strategy, but I'll save that for another post.

Citrix Storms the Beach

Citrix hopes to shake things up here in Cannes by announcing a free version of XenServer, available in late March. The company claims the free product will include management features, resource sharing, and live virtual machine migration, making it the latest run at giant VMware's Virtual Infrastructure suite. 

The vendor also announced Essentials for XenServer and Hyper-V, a suite of enterprise-targeted management components for lab automation, dynamic provisioning and storage integration, all blessed by Microsoft in an alliance release.

Details are sketchy, but Citrix is obviously seeking to tap into the economic constraints it expects buyers to face in 2009. A free enterprise-class server virtualization platform might be the teaser hesitant buyers need to keep virtualizing in a tough year. And, the vendor hopes, it could lay the foundation for upselling advanced management in the future, when budgets loosen.

I'll let you know what I think of this competitive attack later in the week: I'm sitting down with all three vendors here at VMworld. This flurry of competitive news is sure to make some waves, and with that I promise to drop the nautical metaphors for good!

VMworld Europe 2009: Kick-Off!

VMworld Europe 2009 kicks off tomorrow morning in Cannes, France. Taneja Group is on site and will be blogging on the hot announcements, trends, and session highlights all week. As we've seen at other VMworld conferences over the past two years, partners and competitors alike will leverage the buzz generated here to reach a wider audience for their messages, from new releases to game-changing alliance announcements.

We're expecting exciting developments from the virtual desktop, network, and storage vendor ecosystems, along with a rush of entrants into the virtual environment management and optimization market. Watch this space as we filter through the noise to bring you the best of virtualization technology for 2009!

Top 10 Practices for AFTER the eDiscovery Sale

With more and more eDiscovery processes being brought into the corporation, this is a good opportunity to remind eDiscovery vendors what to do after the sale – and what not to do. Note: I came out of IT and was in systems support for over a decade. Believe me, I know how customers can be. They still pay your bills.

1. Make sure your customer likes your support engineers as well as they liked your sales guys. No one expects the support people to be the charmers the sales reps usually are, but they don’t want Android Boy either. (Or Android Girl.)
2. Sit down with the customer and assign real dates to the deployment schedule. Stick to the real dates on the deployment schedule. When the customer takes too long to do something – and they inevitably will – adjust the schedule. When the project ends up being over time and over budget, guess who the customer will blame? That’s right, you. Make sure you are there, are available, and take real responsibility for the schedule.
3. Document milestones, especially important in a complex project. Remember point #2.
4. Don’t try inventing the feature that you promised the customer you had, but really didn’t. Presumably you didn’t promise them that in the first place, no matter what your boss said. If you did, I can’t help you.
5. Make sure your customer understands that most eDiscovery deployments are a pretty big deal. Both technology and processes are changing, and even in pilot deployments there is a big learning curve. Be there with training and hand holding. This is especially critical for the lawyers, but even IT needs to know what is available to them in the technology. “Just get it to work” is their battle cry, but you also want them to realize your product’s real advantages while keeping it simple to manage.
6. And speaking of learning: for heaven’s sake, don’t charge support hours for one of your people to learn your own technology. The customer environment and infrastructure, sure. Your own product, no.
7. Your support people need to understand not only the product, but also the ramifications of different types of matters. This doesn’t require that they be attorneys or litigation support specialists, but does require that they know the concepts and stages of legal eDiscovery and/or compliance workflows.
   
8. If the customer likes you and your product, do a customer case study or solution brief. Even with lots of companies going unnamed these days, you can almost always work up an unnamed customer scenario. Prospects love to see the proof in the pudding, and real-world examples provide that nicely.
9. Once deployment is a success, keep up with the customer and make sure they are happy. Even a matter as small as switching account reps can make or break the next upgrade or expansion project.
10. Congratulate yourself and your people for a job well done. It’s a tough world out there, and – with apologies to the vegans – you brought home the bacon. Now rev up the engines and go do it again.

Thoughts on Source Based Storage Capacity Optimization

I've had the chance recently to spend a lot of time talking to vendors about their source-based storage capacity optimization (SCO) options. In short, these are generally data de-duplication + compression technologies targeted for use with secondary storage that are done using host (i.e. backup client) resources so that the backup gets into capacity optimized form before it gets sent out across the WAN. The big benefit: ROBO backups can usually be done in a fraction of the time since you're sending a lot less data, and it offers similar advantages during restores (although the fact that most restores are at the object level means that the technology shines operationally more for backups than restores). The big knock: it takes host cycles, which can really affect application performance when backups are being done on-line.

Most of the majors have an offering in this space. EMC has integrated Avamar technology into the NetWorker client, Symantec offers PureDisk technology and is able to track data backed up by it in the NetBackup catalog, and IBM and CommVault recently announced source-based SCO technology for their respective flagship backup products (TSM and Simpana). VMware will also be entering the game this month with an interesting twist to the technology. Expect other enterprise backup software players to enter this space as well in the coming months. Target-based SCO options (e.g. storage target appliances like Data Domain's DD appliances, FalconStor's VTL, IBM's TS7650 (Diligent), Sepaton's S2100, etc.) are outselling source-based options not quite 10:1, but choosing the technology that's right for you depends on what your problem is.

What has struck me about these source-based offerings is that there are significant differences between them that really affect how much backup client overhead they impose. The whole reason to look at SCO technology in the first place is generally to reduce the amount of data that has to be handled, whether it's for WAN savings or better managing storage capacity in these times of explosive data growth, so capacity optimization ratios achieved are important. The solutions that tend to achieve the highest ratios usually operate at the sub-file level and leverage a global de-duplication repository (index) of some kind. But if the problem you need to solve is being caused by WAN bandwidth issues, that should drive you to look at source-based SCO options, and we'd encourage you to understand the overhead impact of the different options, since there are real differences. Obviously there are other issues you'll care about as well, such as application compatibility, reliability, cost, and how easy it is to integrate with what you're already doing in data protection. Look underneath the covers on these products to understand how they do what they do, and how that will impact backup client performance in your environment.

Data Domain: Been Around The (Sub) Block A Few Times...

Data Domain has been telling us for years that experience counts for a lot when you're building a storage capacity optimization (SCO) solution. Given that they were the first vendor to ship a pure play product in that space, it's not surprising. If I was running marketing at Data Domain, I'd probably make the same claim. The funny thing is, they may have a point.

The concept of SCO, regardless of whether you use enhanced compression, file-level single instancing, sub-file level data de-duplication, compound object delayering combined with application-specific capacity optimization algorithms, or some other approach, is unassailable in theory. Buy a disk array, add SCO technology, stir until well blended, and get 10x - 20x the storage capacity. If it works, as an end user why wouldn't you do it?

Well, cost issues aside, it really does have to work. Or at least you have to be able to get it to work in your environment. Based on our discussions with vendors in 2006, we thought that SCO was going to break out in a pretty big way in 2007. In 2008, when we looked back at 2007 revenues in this space, they were still quite small, with many vendors delaying introduction of their solutions for various reasons. The technology did deploy at a rapid rate in 2008, with literally thousands of production installations across all vendors even if you don't count the 3000+ or so that NetApp shipped as part of their FAS environment (who knows how many of them were using it?). But even throughout 2008, we were hearing too many tales from end users about operational difficulties using SCO, spanning everything from performance to data loss to compatibility issues. We've encouraged end users to delve into these issues as they evaluate SCO solutions, and suggested to vendors that there are certain baseline capabilities necessary to enter this space.

Looking at last year, a lot of SCO vendors introduced valuable features such as replication, clustering and global repositories, and high end single stream throughput numbers that topped 1GB/sec. Data Domain had none of those, but they had a product that worked and was consistently reliable. The proof is in the pudding: they ended 2008 with close to 4000 customers and revenue of $274M, representing 122% growth year over year from 2007. They own the #1 spot in data de-duplication based on our "Next Generation Data Protection Emerging Markets Forecast", published in September 2008. Maybe there is something to their "experience" claim after all...