Schlagwort-Archiv: Storage

Open-Source SAN – Part 4: Solaris-based Appliances

There are two open-source storage appliances that are based on Oracle's Solaris: EON and NexentaStor. Additionally you could use OpenIndiana, which is based on illumos, a fork of the former OpenSolaris operating system.

Oracle is still developing Solaris, but there is no open-source version available anymore. OpenIndiana is the community driven version now, based on the still released Solaris code.

As I will stay with OpenFiler for now, I haven't been able to install and test these appliances. Therefor I will also not go into much detail as I don't know enough about them.

Still I wanted to mention them as it seems sometimes like everyone only knows FreeNAS and OpenFiler. But there are good alternatives if you feel unhappy with those two.

 

Linux-based

Solaris is based on UNIX, just as BSD and Linux are. This makes them comparable, still it is not the same. Additionally there are several different BSD descendants and hundreds of distinct Linux distributions. So every single of these products has its right to exist and also advantages and disadvantages depending on the use case.

This makes it impossible to say if one is better than the other or which one you should use. Personally, I'm just more comfortable with Linux and am much more experienced with it.

Also I found that there is more support available on the Internet as it is very widely-used. So even if they have the same root, Solaris is not Linux-based.

 

ZFS

As already mentioned, ZFS was originally developed by SUN Microsystems and was therefor instantly integrated into Solaris. So you can use ZFS natively on EON, NexentaStor and also OpenIndiana and it will work like a charm.

But even if you won't have similar issues to using ZFS on Linux (which is said to be unstable), it is still recommended to have fairly new hardware with a lot of RAM and a fast 64-bit CPU. It would also work on my old servers with only 4 GB of RAM, but I would not really benefit from ZFS rather than running into light performance issues.

 

iSCSI

EON and NexentaStor have both iSCSI support integrated, even if NexentaStor has only an initiator. Still it is quite easy to activate this functionality via the CLI.

As already mentioned, I'm not really familiar with package management on Solaris, so I don't know how it works and which packages are necessary. Still I think it should be easily possible to just install them on OpenIndiana and therefor make iSCSI also work there.

 

Clustering

Solaris is made to be a clustering operating system by default, so it should be working fine. You also don't even need additional packages like Corosync and Pacemaker on Linux but just a class of systems. It could also be possible that clustering is only available for enterprise versions and not in community editions.

But what I know is that DRBD is not available for Solaris. As an alternative there are clustered file systems, even I don't know if they exist on Solaris or not. Writing about that I would really like to find this out and test and play with it a while.

As soon as my small cluster is running stable, I will install VMs and check out Solaris a while. I'm really interested into this now. Additionally, if nothing else works, it is still possible to use Xen for doing the replication with Corosync, Pacemaker and DRBD but still store your data on ZFS/Solaris. So you would use the disks inside your XCP hosts instead real shared storage SAN, still you would prevent a SPoF.

 

Web interface

EON has no GUI at all, NexentaStor has a web interface and OpenIndiana can be installed with a desktop just like any other operating system. This might be a bit of overload for a storage appliance, so you might be better off by using Webmin instead. Or you would have to abdicate a GUI and just do anything via the CLI.

The web interface of NexentaStor looks very good as it shows detailed performance measures and the like. Still this could be a bit overloaded and therefor difficult for beginners. Also there seem to be endless configuration options available, so great for people who want to control just anything but too complicated for everyone else.

 

S.M.A.R.T.

I haven't tested any of these appliances yet so I just don't know anything about their monitoring abilities. As already mentioned, NexentaStor seems to be very good in this.

first: Part 1: Introduction previous: Part 3: OpenFiler

Open-Source SAN – Part 3: OpenFiler

Linux-based

OpenFiler is indeed based on Linux, but unfortunately that is not as good as it could be. It runs on a distribution called Foresight Linux which is again based on rPath Linux, a commercially driven project by a company using the same name.

DistroWatch.com says about this distribution:

rPath Linux, built with the Conary distributed software management system, is not only a distribution in its own right, but also a base technology explicitly designed to enable you to create purpose-built operating system images using the rBuilder Online technology.

This makes clear why OpenFiler runs on this distribution and it may be nice for the developers to build their own OS with less effort. But for users and community developers it is a huge disadvantage.

As already mentioned, rPath Linux is a commercial project financed and organized by a corporation. It is still free (even I don't know if it is also open-source), but there seems to be little to no active community behind it, it is not well known and it is hard to get help online.

Additionally, Conary may be a nice package manager, but in my opinion (and I'm not the only one thinking so) there are far better tools for this. I got quite used to it now, but this took me weeks of mostly frustrating effort to get to that level.

The documentation is quite fine, but still it is not the same to read everything in wikis rather than searching forums or reading mailing lists. You are very likely to find an answer to your questions on the OpenFiler community rather than by Foresight Linux or rPath.

Unfortunately, rPath is not the only company behind OpenFiler, but OpenFiler itself is also commercially oriented. It is still free and open-source and a corporation behind such a project could also be a huge advantage. Not so for OpenFiler.

Support in the forum is very rare, mostly you can read about complains that OpenFiler is not working as it should, that there is no help from the community or that you won't find OpenFiler developers or officials. Reading this can really be frustrating, but still I think that OpenFiler is a great product. Nevertheless you should be an advanced Linux user and you should not expect an absolutely stable appliance where everything is working correctly.

Even if OpenFiler is not made to be installed on an USB pen drive like FreeNAS, it is still possible and also working very fine. The huge advantage is the same. But in comparison to FreeNAS it is much easier to install additional packages on OpenFiler. Or lets say it would be much easier if it, again, weren't rPath with the complicated Conary repository manager and very hard-to-find software builds.

Nevertheless I have to admit that OpenFiler comes very complete just out-of-the-box, which means that there is almost anything you can imagine right there with a fresh installation. Also you can keep these packages up to date with Conary or even via the web interface.

Unfortunately the release cycle of OpenFiler is not regularly nor is it really current and there is not even a roadmap. The community got very frustrated with the release of version 2.99.1 in April 2011, while it was promised to be a major 3.0 release in the end of 2010. This candidate has been postponed several times to Q1 2011 to the end of 2011, now it is Q1 2012 and there is still no OpenFiler 3.0 available.

Now it is said that OpenFiler 2.99.1 with all updates is 2.99.2 which should be exactly the same product as the upcoming 3.0, but only with the old web interface. Still it seems that 2.99 is a beta release that raised lots of errors, issues and bugs that just stay unsolved and even uncommented. There are only very few people left who still believe that OpenFiler is really under active development.

 

ZFS

As already mentioned, OpenFiler comes with lots of packages out-of-the-box. This also includes the zfs-on-linux release which allows creating RAIDz pools and datasets just as it is possible on FreeBSD.

I think there are very less people recognizing these great news so far, so nobody has really tested this on OpenFiler yet. I did and it seems to be working somehow, but not really well. Additionally it is said that these packages should be considered to be developmental and testing rather than stable.

Therefore you are really recommended not to use it in production and critical environments. This is too bad, I would have loved to use ZFS on OpenFiler, but it's more important to have a stable working storage solution rather than having a testing playground.

On the other hand it is actually no such big deal. I thought a lot about using ZFS for my SAN, but at the moment I'm not so sure about this anymore. I want to share my disks via iSCSI as one single LUN which will then host several virtual disk images (VDIs), each providing one VM.

This will compensate most of the advantages of ZFS, so it doesn't make so much sense anymore. The mots promising opportunities for me were RAIDz and snapshots.

  • Still I think that LVM2 on Linux with a software RAID 6 is also very fine and probably much more stable.
  • And then there are snapshots. Actually, snapshots are only great for a somehow “traditional” file system, meaning that you store files on it directly rather than providing VDIs. There is no real application you could benefit from restoring a snapshot of the whole system state when you have running several VMs on it. This would just reset all your machines at the same time to some specific restore point. Therefor it will be better to just do snapshots of the VMs themselves via the hypervisor and running “classical” backup strategies inside the VMs.

Besides zfs-on-linux there is something similar under development – the B-tree file system (Btrfs). This seems to be quite identical to ZFS but is also considered to be not stable yet.

 

iSCSI

I was able to set up an iSCSI target on OpenFiler on top of an “iSCSI block device” and successfully installed a VM on it. The process is quite intuitive and straight forward, just a few clicks and there you go. Also it was possible to use “write-thru” mode which might be somehow comparable to “pass” on FreeNAS (which didn't work there).

What I have not done (yet) are some benchmarking tests for file input and output (Input/Output). Some people claim that “blockio” (which I'm using too) is not quite fast and that you are better off with using “fileio” instead. Also I didn't test any read or write speeds. Luckily this doesn't really bother me, as the installation of the VM was as smooth and fast as normal and I have no real bottlenecks experienced yet.

MPIO

I didn't have the same problems with link aggregation (actually called "bonding" on Linux) on OpenFiler as I had with FreeNAS. But this is mainly because I simply wasn't seduced to do so, because I'm running it inside a cluster. As the heartbeat needs its own dedicated network connection, I had to reserve one of my only two network interface cards (NICs) for this. So nothing left to bond.

This will also prevent MPIO, but it may be still working somehow. I'm not really sure about this (yet), but while setting up only one iSCSI target on the active cluster master node, I can still see two targets on my initiators (as long as both nodes are running). But on the other side when checking the connections of my targets, I can see two of them, but only one from each XCP host. So probably no MPIO.

I will have to do some further testing on my XCP machines, as I configured them to use MPIO and it just worked without any error messages. But while trying to probe the storage repository (SR) for multi-pathing, it gave me an error with using iscsiadm. It didn't say that it can't find multiple targets, but only that the service is not able to execute because a file is missing. So I will give this another try when time allows me to do so.

Also I will test the reactions of sudden disconnections then, to figure out if redundancy is working or not. This is more important for me than load-balancing, as my VMs will not have such heavy traffic. Still I would be happy to have this feature, too.

 

Clustering

Clustering on OpenFiler works like a charm. OK, it is not provided by default as they want you to pay for commercial support to activate the services. Nevertheless you can just follow this simple how-to and make it work!

OpenFiler uses the great free and open-source Linux tools Corosync, Pacemaker and DRBD to enable an active/passive hot-standby master/slave cluster. DRBD can be considered as a RAID 1 mirror, just via the network between two servers instead of two disks inside one server.

Therefor you can only use a cluster size of two nodes, but this is just fine for me. All I wanted to have is to prevent the classical SPoF with shared storage, as this just vanishes the actual advantage of that principle.

There will be at least two hypervisors running, so if one host fails, the VMs get migrated to the other node automatically and the clients don't even recognize that there was an actual system failure. But what if the shared storage is only provided by one single server and this one fails?

The whole cluster crashes and you again lose your “high availability” functionality. But with Pacemaker and DRBD the two storage nodes keep in sync all the time on a deep file system basis, so if the active master node fails, the passive slave node can take over all the services immediately without any data loss. This is why it is called "hot-standby" in comparison to a "cold-standby" solution as provided by FreeNAS and rsync.

 

Web interface

The web interface looks nice, only this self-signed security certificate thing sucks when using Internet Explorer. Actually, if you think of OpenFiler as an appliance, it is only a collection of tools and a GUI to handle these. So it should be also possible to set up your own storage server with, let's say, Ubuntu and just copy the files from OpenFiler to use its web interface.

I think the GUI of OpenFiler is really OK. When you use it some time you soon get used to it and it becomes quite comfortable. As mentioned for FreeNAS already, these GUIs are all not very user-friendly or intuitive to use, but still this one is as good as it could be.

The only thing is that there are really a lot of things that just don't work via the web interface or are simply not available. So you actually have to use the CLI a lot.

A few examples:

  • Installing updates on a clean fresh install raises numerous issues and can even crash your system, so you can only install some of them from the GUI and the rest via the CLI.

  • Partitioning does just not work via the web interface, so you have to use parted commands.

  • There is a menu for clustering, but it is not available as it only says that the service is not running. Of course you're not able to start it. So you have to configure everything via the CLI, but as soon as you're finished, you can at least see the settings inside the GUI, even if you are not able to change anything.

So I think if you are already familiar with Linux and storage tools and protocols, you may be better off with using the command line only and just forget about a GUI. If you are an absolute beginner, you will also have difficulties with this web interface, but at least you should be able to get what you want. With little help from the community, it should also be no problem to type simple commands, just for the case that you have an issue which cannot be solved via the GUI.

 

S.M.A.R.T.

S.M.A.R.T. is also available for OpenFiler via the smartmontools package. I have tested some simple smartctl commands and everything seems to be working correctly. It is not as convenient as for FreeNAS, which comes with full S.M.A.R.T. functionality pre-built.

For OpenFiler you would have to configure everything yourself and use cron and sendmail to enable email notifications. I just had not the time to set this all up yet, but am planning to do so soon. Of course I will let you know how it works.

I don't use SNMP so I also don't know if it is available for OpenFiler. What it indeed has are email notifications for RAID events and this is working like a charm. It will let you know when and which drive fails and after you replace it, you will get a rebuild progress message regularly.

There are also some simple networking and storage statistics available, but these are not sent via email automatically. As mentioned several times already: this is Linux, so you are very welcome to set up everything as you like!

first: Part 1: Introduction previous: Part 2: FreeNAS last: Part 4: Solaris-based Appliances

New Wiki for Storage and Virtualization

I have updated my Docs section and changed from DokuWiki to MediaWiki. This was just because MediaWiki is much more state-of-the-art, has better functionality and is easier to maintain.

I will try to create new articles with useful information for the categories of storage and virtualization on a regular basis. This will mainly contain FreeNAS 8 and OpenFiler 2.99 SAN appliances and Xen Cloud Platform (XCP) hypervisor.

In separation to the how-tos in my blog these will be more of a kind of knowledge articles and guides rather than simple step-by-step processes. I really hope that this will help some people to get solutions for the issues they were looking for.

For now there are the following articles online:

Enjoy reading and please let me know when the information you found there was somehow useful for you!

Open-Source SAN – Part 2: FreeNAS

Linux-based

FreeNAS is a very great stable working product, well documented and has a big active and motivated community. By the way, I also contributed my part by doing the German translation for the (yet un)official FreeNAS 8 FAQ. You can find it in my wiki, which will hopefully grow from time to time with useful knowledge articles for the categories of storage and virtualization.

Unfortunately it is not Linux-based, but runs on a specially minimized version of FreeBSD. This minimization makes it fit on a plain USB pen drive you can directly boot it from.

This is a huge advantage, as it allows you to use all your hard disks for storage only and you don't lose space for a system partition. Specially in my case this would be a great solution as I only have old hardware with quite a lot but small disks (6x 320 GB SATA). So I'm very happy for any MB I can use for storage and don't have to reserve for an OS.

But this is also a disadvantage. Not only that FreeNAS is no Linux, it is also very limited due to a maximum size to fit on an USB stick. This makes it quite difficult to install additional packages you might need for your server. There's not even an automatic update function available.

 

ZFS

Still, there is another point that clearly speaks for FreeNAS. ZFS was developed by SUN Microsystems (now Oracle) and soon available for their own operating system Solaris. Some time later, ZFS has been ported to FreeBSD and seems to be working very well on this system now, too. For Linux this is not that easy and ZFS is not yet natively available for any Linux distribution.

So why is ZFS so important for me? It is simply a very promising file system (FS) and logical volume manager (LVM), including data integrity verification against data corruption modes, snapshots and copy-on-write clones, continuous integrity checking and automatic repair and is implemented as open-source software. Additionally, RAIDz offers a great new solution for setting up clever and secure mirrors or arrays.

In my opinion, the biggest advantage are ZFS snapshots, and FreeNAS does not only offer this functionality, but also introduces automatic snapshot and replication tasks. The FreeNAS 8.0.2 User Guide says:

FreeNAS supports the secure replication of ZFS snapshots to another remote FreeNAS system (or any other system running the same version of ZFS and a listening SSH server). This allows you to create an off-site backup of the storage data.

So it's creating snapshots of the whole system regularly however often you want it to and enables you to do a restore of the full system state. And with the replication tasks, you can additionally save the snapshots on another machine to have double the security.

Unfortunately, I've never been able to set up such a replication between two FreeNAS boxes. I'm not affirming that it's not possible, all I'm saying is that it's really difficult and I was just not able to get it up and running. You have to configure SSH keys to enable mutual authentication between the two servers. This is very well documented in the user guide, but although I was exactly following all the steps, it was just never working.

 

iSCSI

I had two FreeNAS boxes running for a while and had the chance to test all the functions I need. I also tested iSCSI. First of all, what do I need iSCSI for?

It is an IP-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and can be used to transmit data over LANs. So it can enable location-independent data storage and retrieval. This makes it a SAN protocol, allowing organizations to consolidate storage into data center storage arrays while providing hosts (in my case virtual machine hosts) with the illusion of locally-attached disks. Unlike traditional Fibre Channel, which requires mostly expensive special-purpose cabling, iSCSI can use existing network infrastructure [Wikipedia].

I was able to set up an iSCSI target on FreeNAS on top of ZFS and successfully installed a VM on it. Still, there were some major issues I was a little bit concerned about. The user guide says:

type of device: choices are disk, DVD, tape, or pass (choose pass in a virtual environment)

under iSCSI Target Settings.

As I want to use the server for VM storage, I chose pass. Unfortunately it is not possible to start the iSCSI service then, which makes the target inaccessible. Therefore I selected disk and well, it still was working, so who cares? You know, I do somehow, if it gives you an option you should but can not choose, this is somehow strange for me.

Multipath I/O (MPIO)

Also it was not easy to set up link aggregation on FreeNAS, I finally got it running after several tries, but had to do it directly via the console, as I lost connection to the web interface. There was another thing I thought that would not be working correctly, but now I know that it was my own fault.

When setting up link aggregation, you're not able to use iSCSI MPIO, as also described in the user guide:

NOTE: LACP and other forms of link aggregation generally do not work well with virtualization solutions. In a virtualized environment, consider the use of iSCSI MPIO through the creation of an iSCSI Portal as demonstrated in section 8.14.6. This allows an iSCSI initiator to recognize multiple links to a target, utilizing them for increased bandwidth or redundancy.

I wanted to have redundancy and load-balancing on network level, but obviously I was just thinking in a wrong direction. So you can use FreeNAS very well for iSCSI and VM storage!

 

Clustering

Besides the fact that FreeNAS is not Linux-based, there is another very big disadvantage for me, why I won't use it for my system. There is just no clustering, high availability or reliability functionality available, except you change to TrueNAS, the commercially developed, non-open-source version of FreeNAS.

What you can use and is also recommended by the user guide, is called rsync:

Rsync is a utility that automatically copies specified data from one system to another over a network. Once the initial data is copied, rsync reduces the amount of data sent over the network by sending only the differences between the source and destination files. Rsync can be used for backups, mirroring data on multiple systems, or for copying files between systems.

So that says what it is all about - a passive backup solution. It's great and very useful though, but you still have the problem of a single point of failure (SPoF). So if your first FreeNAS box crashes, you would still have your data on the other one, but would have to reconfigure all clients to access the new server now. So it's more a kind of a passive standby that needs some manual work to get it up and running.

 

Web interface

The web interface looks nice, but is not really working with Internet Explorer, as there are some major display errors which make it impossible to configure everything as you want. This is no big deal, in Firefox it is working just fine.

Occasionally, a GUI would not really be required, but as I'm not that Linux and console pro (yet ;-)) it just makes my life much more easy. Also it enables you to do simple settings much quicker when it only needs two clicks rather than typing config files and complex commands. Also I have to admit that the interfaces of the appliances I have tested so far are not really intuitive. So you need some time to get comfortable with them, too.

 

S.M.A.R.T.

What is S.M.A.R.T and what do I need it for? Wikipedia says:

S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology) is a monitoring system for computer hard disk drives to detect and report on various indicators of reliability, in the hope of anticipating failures. When a failure is anticipated by S.M.A.R.T., the user may choose to replace the drive to avoid unexpected outage and data loss.

This makes it a great tool to monitor your disks for temperature, corrupted cylinders and so on. It will automatically send email notifications when preconfigured limits are reached or other problems are detected. It doesn't necessarily have to be S.M.A.R.T., FreeNAS also offers SNMP and email notifications for RAID events, but also networking and storage statistics.

first: Part 1: Introduction next: Part 3: OpenFiler last: Part 4: Solaris-based Appliances

Open-Source SAN – Part 1: Introduction

The last month has seen very less blogging. This is due to several reasons like major changes in my private free time (but I don't want to bore you with that) and busy weeks at work (our mail server had a total system fail). But mainly it is because I'm working on a new project which takes all of my time and attention at the moment.

I'm evaluating the best working solution for an open-source Storage Area Network (SAN) that provides a small private cloud (or at least a cluster). As it took me hours of research, days of testing and numerous issues I was running into, I thought I should share my collected experience with you.

Maybe this could help you with your own decision and save you from wasting so much time as it was in my case. First of all I want to make clear that I'm not really going to compare different solutions with each other, but more cite what is available out there and might fit you best.

The main “problem” is that all of the mentioned products are very good software and have their own advantages for special use cases. So at last you have to make your own decision about what you exactly need and then find the solution that fits you best.

As this is also what I did, let me mention what were my requirements:

  • Linux-based
  • ZFS or at least snapshot functionality
  • iSCSI
  • Cluster / replication functionality, high availability or reliability (DRBD)
  • Web Interface or some other kind of GUI
  • S.M.A.R.T or other kind of monitoring and automatic notification system

Now the solutions I found:

If you have something to add here, I would appreciate if you could let me know via a comment to this post. I will discuss my personal requirements within every single appliance in more detail in the following blog posts.

next: Part 2: FreeNAS last: Part 4: Solaris-based Appliances