12th Discussion-21 May 2012 - Revision history

Hemant Kelkar: /* Topic 1 Managing Storage in a Core Facility */

2012-05-24T11:43:32Z

Topic 1 Managing Storage in a Core Facility

Sexton at 13:41, 22 May 2012

2012-05-22T13:41:21Z

Sexton at 13:38, 22 May 2012

2012-05-22T13:38:15Z

Sexton at 13:36, 22 May 2012

2012-05-22T13:36:31Z

Sexton at 13:28, 22 May 2012

2012-05-22T13:28:58Z

Simon andrews: /* People Present */

2012-05-22T11:06:38Z

People Present

Simon andrews: /* Transcript of Minutes */

2012-05-22T08:37:46Z

Transcript of Minutes

Simon andrews at 14:53, 21 May 2012

2012-05-21T14:53:02Z

Simon andrews: Created public call page.

2012-05-09T15:41:15Z

Created public call page.

New page

= Brief Description and Continuing Discussion: =

This discussion will have one main topic for discussion, followed by an open floor where people can raise other topics of interest.

== Topic 1: Managing Storage in a Core Facility (Led by Simon Andrews) ==

With ever larger datasets being produced the management of data storage in a core facility is a constant concern for many people. This topic will aim to cover all aspects of storage, from looking at the the physical systems people use to how data is managed, shared with users and charged for.

== Preliminary Information ==

Questions which would be relevant to this topic might include:

* Are core facilities the right people to be managing storage

* What systems are people using to store and back up their data

* What data are people storing?

* How are you managing your storage? How do you ensure you're only storing what you need and making sure data which is no longer useful is deleted?

* How do you make stored data available to users? What access controls do you require? How do you allow users to share data both internally and externally?

* Do you charge for storage, and if so how?

* How do you integrate local storage with large public datasets?

== Topic 2: Open Floor ==

After the main discussion there will be an opportunity to raise other topics which might be of interest to the community. These can be suggested in advance below, or raised on the call itself.

== Suggested Topics ==

* Add topics here...

=Transcript of Minutes=

==People Present==

== Topic 1 Managing Storage in a Core Facility ==

@@ Line 80: / Line 80: @@
 [FL] has a shared server for scientists where they can access and modify their data.  They don't back this up at all and scientists must copy it themselves if they want to back up. At the end of a project all data is wiped after 3 months.
-[??] has a patron model with shared space on a common infrastructure.  People have the option of buying their own disks within the infrastructure. Duplication is a problem.  Storage is done for 3 years by default but managing data older than that has proved difficult and in effect the storage may end up being indefinite.
+[HK] has a patron model with shared space on a common infrastructure.  People have the option of buying storage within the infrastructure. Duplication is a short term problem.  Archival storage (tape backup) is done for 3 years by default. In effect the tape backup storage may end up being indefinite.
 [BR] Partners has 3 levels of storage.  IT have a managed Netapp system but this is very expensive and is too much for most scientists.

@@ Line 84: / Line 84: @@
 [BR] Partners has 3 levels of storage.  IT have a managed Netapp system but this is very expensive and is too much for most scientists.
-For clusters they have storage based on iBricks which allows for different storage tiers and can be accessed by NFS or Samba.  Finally use compellant storage for working space.  Allows for granular authorisation.  Works well.
+For clusters they have storage based on iBrix which allows for different storage tiers and can be accessed by NFS or Samba.  Finally use compellant storage for working space.  Allows for granular authorisation.  Works well.
 Partners provide some storage attached to the HPC cluster but this is temporary storage space and is quickly deleted.  Users have to pay for longer term storage.  Have moved away from tape for archiving as it's too expensive and has too many changes.  Now using a home-built system based on commodity hardware.  This is a series of locally built storage units linked together using gluster.  They used to have Sun Thumpers as the underlying storage but are moving away from these.  Transition hasn't been simple as they were using ZFS under gluster which wasn't officially supported.  Have 800TB under RAID6 and have found this to be manageable.

@@ Line 114: / Line 114: @@
 [??] Said that their data used to go to Thumper systems but now goes direct to isilon storage and is copied from there to its final destination.
-[SA] Asked if anyone had looked at using data compression in their storage solutions?  Particularly interested to know if anyone had tried out CRAM?
+[SA] Asked if anyone had looked at using data compression in their storage solutions?  Particularly interested to know if anyone had tried out [http://www.ebi.ac.uk/ena/about/cram_toolkit CRAM]?
 [DS] Said they'd tried CRAM at Baylor.  It worked but the tools provided were not really working well enough yet.

@@ Line 90: / Line 90: @@
 [MT] At the Broad they don't believe in long term storage within the facility. The group charges for 3 years worth of storage as part of the sequencing charge.  They make every effort to make the data available directly to users rather than having them copy it to other storage systems.  This keeps the bulk of the storage on the common central system and the users create smaller derived files such as variant files which are much smaller.  After 3 years the raw data is deposited into NCBI and can be removed from the local storage.  All storage is billed as a service and measured in terrabyte years.
-Access to data is managed by a custom metadata management system which integrates with their primary LIMS.  This then serves as the interface to the data which can be accessed through a web interface, web services or a FUSE based filesystem mount.  The FUSE mount is present on all compute nodes on their cluster and provides managed access to all data.  The service picks up the access control rules from the data management system and provides a custom view of the run data to each user.  Scales much better than using traditional file system permissions to do this.  Normal permissions aren't flexible enough to handle the complex rules which might be present (eg limit of 16 group memberships per user on NFS).  The FUSE overlay does entail a performance hit but is practical.  The underlying technology is simply an NFS mount to an isilon cluster.   The FUSE system is effectively creating a series of links into the main data store.
+Access to data is managed by a custom metadata management system which integrates with their primary LIMS.  This then serves as the interface to the data which can be accessed through a web interface, web services or a [http://fuse.sourceforge.net/ FUSE] based filesystem mount.  The FUSE mount is present on all compute nodes on their cluster and provides managed access to all data.  The service picks up the access control rules from the data management system and provides a custom view of the run data to each user.  Scales much better than using traditional file system permissions to do this.  Normal permissions aren't flexible enough to handle the complex rules which might be present (eg limit of 16 group memberships per user on NFS).  The FUSE overlay does entail a performance hit but is practical.  The underlying technology is simply an NFS mount to an isilon cluster.   The FUSE system is effectively creating a series of links into the main data store.
 Same system is used to share data with collaborators - usually they are given access to the analysis cluster so can access the data the same way local researchers do.
-Also looking at integrating IRODS into this system.  This is a descendent of the storage resource broker system.  It is a metadata management system which is based on a key value store, but also provides a series of tools to move and manage the data.  It uses micro-services to do things like delete, copy or change the data and you can use these to construct complex business rules which are based on the metadata stored in the system.  It's not likely to be useful as the main store of experimental metadata due to the relatively unstructured nature of the data, but it might make sense to copy metadata from a structured LIMS into IRODS.
+Also looking at integrating [https://www.irods.org/ IRODS] into this system.  This is a descendent of the storage resource broker system.  It is a metadata management system which is based on a key value store, but also provides a series of tools to move and manage the data.  It uses micro-services to do things like delete, copy or change the data and you can use these to construct complex business rules which are based on the metadata stored in the system.  It's not likely to be useful as the main store of experimental metadata due to the relatively unstructured nature of the data, but it might make sense to copy metadata from a structured LIMS into IRODS.
 Since storage is now becoming the limiting part of many pipelines some vendors are now looking at integrating IRODS into storage controllers so data processing can happen closer to the disk.  Would allow, for instance, the conversion of BAM files to FastQ on the fly in the storage controller.

← Older revision		Revision as of 13:28, 22 May 2012
Line 107:		Line 107:

	[DS] Said they were putting together a 200TB BlueArc system for the instruments to write to.		[DS] Said they were putting together a 200TB BlueArc system for the instruments to write to.
		+
		+	[MT] Said their sequencing instruments were writing to one Isilon cluster, then primary analysis was writing to another cluster, and finally tertiary analysis was writing to yet another Isilon cluster.

	[??] Said that they were using the isilon SyncIQ system to replicate data from the initial storage to the final storage system.		[??] Said that they were using the isilon SyncIQ system to replicate data from the initial storage to the final storage system.

@@ Line 50: / Line 50: @@
 * Madeline Gogol
 * Stuart Levine
 == Topic 1 Managing Storage in a Core Facility ==

@@ Line 37: / Line 37: @@
 ==People Present==
 == Topic 1 Managing Storage in a Core Facility ==
-Charlie Whittaker kindly provided [[12th_Discussion-21_May_2012_Charlie_Whittaker | some information]] about his facility to help answer some of the questions posed.
+Charlie Whittaker kindly provided [[12th_Discussion-21_May_2012_Charlie_Whittaker | some information]] in advance of the call about his facility to help answer some of the questions posed.

← Older revision		Revision as of 14:53, 21 May 2012
Line 39:		Line 39:

	== Topic 1 Managing Storage in a Core Facility ==		== Topic 1 Managing Storage in a Core Facility ==
		+
		+	Charlie Whittaker kindly provided [[12th_Discussion-21_May_2012_Charlie_Whittaker \| some information]] about his facility to help answer some of the questions posed.