ISMB 2016: BioinfoCoreWorkshop - Revision history

Ybukhman: /* Big Data */

2016-07-20T17:20:30Z

Big Data

Mgogol: /* Big data */

2016-07-19T18:36:42Z

Big data

Mgogol: /* Big data */

2016-07-19T14:04:00Z

Big data

Mgogol: /* bioinfo-core Dinner */

2016-07-18T14:33:30Z

bioinfo-core Dinner

Mgogol at 14:30, 18 July 2016

2016-07-18T14:30:48Z

Mgogol: /* Discussion with notes */

2016-07-18T14:25:55Z

Discussion with notes

Mgogol: /* Big Compute */

2016-07-18T14:04:50Z

Big Compute

Ssayols: /* Big Compute */

2016-07-14T12:19:33Z

Big Compute

Bgrichter: /* Big Data */

2016-07-12T19:09:24Z

Big Data

Bgrichter: /* Big Data */

2016-07-12T19:08:39Z

Big Data

@@ Line 75: / Line 75: @@
 ==Discussion with notes==
 ===Big Data===
-Yury Bukhman.  The GLBRC consortium consists mainly of people at U of Wisonson and a small group at Michigan state university.  The consortium in mainly involved in agriculture and sustainablility.  Very practical--loking to develop biofules and biochemicals.
+Yury Bukhman.  The Great Lakes Bioenergy Research Center, GLBRC, is based at U of Wisconsin - Madison and Michigan State University.  It also includes a few groups at other institutions. The informatics core consists of a larger group at UW and a smaller one at MSU. The Center is involved in all aspects of biofuels research, including agriculture and sustainability, biomass processing, and microbial fermentation of biomass-derived hydrolysates to produce fuels and chemicals.  The goal is develop basic science that will enable economical and sustainable production of biofules and industrial biochemicals in the future.
-All groups in the consortium is mandated to have a data management plan that's reviewed by the bioinformatics core on a yearly basis.  This provides a consulting opportunity for bioinformatics planning and research IT: both on prem resources as well as cloud services.
-The core has developed  a metadata database called GLOW.
+All groups in the Center are mandated to have a data management plan that's reviewed by the informatics core on a yearly basis.  This provides a consulting opportunity for bioinformatics planning and research IT: both to assess the needs of the Center and to suggest existing services to group leads.
 Alberto Riva: The bioinformatics core facility sits within a closely organized group of core facilities that can be used for Life and Health Science.  Additionally, they have access to large and shared IT resources with segments setup for their specific use cases (a private area of the large 10,000 core cluster, for example).
@@ Line 88: / Line 91: @@
 At Purdue, there is not a single big large set of data, but 1000's of small datasets.  Purdue core works with users who have varying levels of analytic and IT knowledge. They find that they have spend time working on datasets in order to adapt/format/clean them for analysis as well as understanding the experimental parameters.  Not everyone knows what goes on inside and behind the scenes of the core in performing this work.  They expect the work to be quick, but without prior involvement in developing the experiment, it takes days to get the dataset to a state where it can be run through their analysis!  Educating the students and educating their users about the data, dataset and the analysis is important.
-Collecting metadata of small and large datasets is a big problem, particularly if one wants to combing data across experiments or in the future. It is required to compare different datasets.  Additionally, when submitting new data to public datasets, the repositories require long list of metadata.  GLBRC maintains a spreadsheet that's required to be filled out that specifically focused on the metadata.  This forces investigators to think about the metadata.
+Collecting metadata of small and large datasets is a big problem, particularly if one wants to combing data across experiments or in the future. It is required to compare different datasets.  Additionally, when submitting new data to public datasets, the repositories require long list of metadata.  GLBRC maintains LIMS systems and a metadata spreadsheet that's required to be filled out to submit datasets to its GLOW omics meta-database.  The latter is similar to submitting to GEO and other public repositories. It encourages investigators to record their metadata at an early stage.
 Biggest challenges for Riva is in educating users on how to generate the data--you may have all the big data you want, but if the experiment is not designed properly, there's quite a lot of cruft.

← Older revision		Revision as of 18:36, 19 July 2016
Line 29:		Line 29:
	The Bioinformatics Core of the ICBR provides bioinformatics services to the large and diverse scientific community of the University of Florida. Routine handling of projects covering a vast spectrum of biological and biomedical research requires a flexible and powerful data infrastructure. Implementation details of a software development environment (Actor) for reliable, reusable, reproducible analysis pipelines will be discussed, as well as insights on managing big data projects in a core setting.		The Bioinformatics Core of the ICBR provides bioinformatics services to the large and diverse scientific community of the University of Florida. Routine handling of projects covering a vast spectrum of biological and biomedical research requires a flexible and powerful data infrastructure. Implementation details of a software development environment (Actor) for reliable, reusable, reproducible analysis pipelines will be discussed, as well as insights on managing big data projects in a core setting.

−	[[File:~~Alberto_updated~~.pdf]]	+	[[File:Riva-WK03-ISMB16.pdf]]

← Older revision		Revision as of 14:04, 19 July 2016
Line 29:		Line 29:
	The Bioinformatics Core of the ICBR provides bioinformatics services to the large and diverse scientific community of the University of Florida. Routine handling of projects covering a vast spectrum of biological and biomedical research requires a flexible and powerful data infrastructure. Implementation details of a software development environment (Actor) for reliable, reusable, reproducible analysis pipelines will be discussed, as well as insights on managing big data projects in a core setting.		The Bioinformatics Core of the ICBR provides bioinformatics services to the large and diverse scientific community of the University of Florida. Routine handling of projects covering a vast spectrum of biological and biomedical research requires a flexible and powerful data infrastructure. Implementation details of a software development environment (Actor) for reliable, reusable, reproducible analysis pipelines will be discussed, as well as insights on managing big data projects in a core setting.

−	[[File:~~alberto~~.pdf]]	+	[[File:Alberto_updated.pdf]]

← Older revision		Revision as of 14:33, 18 July 2016
Line 133:		Line 133:
	Some members met for dinner at Garden Grove... Including a new honorary member...		Some members met for dinner at Garden Grove... Including a new honorary member...

−	[[File:~~goofy~~.jpg~~\|300px\|thumb\|left\|Goofy~~]]	+	[[File:goofy_small.jpg]]

← Older revision		Revision as of 14:30, 18 July 2016
Line 128:		Line 128:

	When using the cloud, don't you still assume you will have to store input / output data locally? Someone will download the data. End user can download data to analyze. Sergi: We generate tons of data, don't want to move it to Amazon... J: We think there will be a hybrid model. For storage, 1 Terabyte is 90$ per year, but that includes maintenance. Different options - S3, Glacier (70$ per Tb per year). There will be a hybrid solution for cloud and local compute and storage. The cloud can expand the possibilities of what we can do.		When using the cloud, don't you still assume you will have to store input / output data locally? Someone will download the data. End user can download data to analyze. Sergi: We generate tons of data, don't want to move it to Amazon... J: We think there will be a hybrid model. For storage, 1 Terabyte is 90$ per year, but that includes maintenance. Different options - S3, Glacier (70$ per Tb per year). There will be a hybrid solution for cloud and local compute and storage. The cloud can expand the possibilities of what we can do.
		+
		+	=== bioinfo-core Dinner ===
		+
		+	Some members met for dinner at Garden Grove... Including a new honorary member...
		+
		+	[[File:goofy.jpg\|300px\|thumb\|left\|Goofy]]

← Older revision		Revision as of 14:25, 18 July 2016
Line 102:		Line 102:

	Bottom line, last thought: a core and the personnel within it has to be adaptable in order to understand what is brought to them. No 2 experiments are alike and needs continuously change. The trends, technology capability and tools change. Need to remain flexible, adapt pipelines, process and people.		Bottom line, last thought: a core and the personnel within it has to be adaptable in order to understand what is brought to them. No 2 experiments are alike and needs continuously change. The trends, technology capability and tools change. Need to remain flexible, adapt pipelines, process and people.
		+
		+	=== Big Compute ===
		+
		+	Do you keep the analysis results on the cloud or download to local storage? J: For the pilot they synced data to S3, then downloaded data back to local storage. Sergi: use a private cluster, the data is there.
		+
		+	Do you pass costs on to users? J: That's the eventual plan, to pass costs. They might not LIKE it... Haven't done it yet, as this was just a pilot.
		+
		+	On the cloud, you can just use the storage you need.
		+
		+	Is data transfer feasible without a big AWS pipe like MIT has? J: Works well for MIT...
		+
		+	How do you schedule the data to be processed in AWS? J: Just use SGE - the same as a local cluster. Make your cluster size 1000 (or whatever) and run your jobs simultaneously. After test is finished, move results to S3.
		+
		+	How many users do you support? Answer for all in the range of 50?
		+
		+	Do your users themselves use docker or amazon? Sergi: No. They will sometimes share pipelines with users, but permissions issues crop up... Sara: Users are not sophisticated, they might have trouble using a command line. J: I set up scripts for them sometimes, no AWS yet. Launching AWS in the cloud is kind of difficult.
		+
		+	How do you address training issues with these new tools? Different solutions for different users. Naive users are not going to use these tools yet, more advanced might start. Depends on them. Sergi: we send them to university training for unix.
		+
		+	Did you try the pilot with different kinds of nodes? How was error handling? J: Yes, tried different nodes. Error handing is no different locally versus in the cloud.
		+
		+	In a distributed docker system, how are errors handled? Sergi: Inside or outside a container, errors are really the same. It's an abstraction layer.
		+
		+	How are you becoming more integrated with the technology group or bringing scientists together with the technology? Many don't care about the technology. They want their raw data and their results or final results. Some groups have their own bioinformaticians - with them cores work closer, provide code, etc. They are a link to technology groups. Many cores have courses for users to learn to do things themselves.
		+
		+	When using the cloud, don't you still assume you will have to store input / output data locally? Someone will download the data. End user can download data to analyze. Sergi: We generate tons of data, don't want to move it to Amazon... J: We think there will be a hybrid model. For storage, 1 Terabyte is 90$ per year, but that includes maintenance. Different options - S3, Glacier (70$ per Tb per year). There will be a hybrid solution for cloud and local compute and storage. The cloud can expand the possibilities of what we can do.

@@ Line 51: / Line 51: @@
 With a variety of computing infrastructures available, building robust, transferable pipelines can increase utilization of compute resources. NGS analysis pipelines implemented as docker containers and deployed on a variety of compute platforms – (cluster, supercomputer, or workstation) will be discussed.
-[[File:Sergi.pdf]]
+[[File:Sergi_updated.pdf]]
 Speaker: '''Jingzhi Zhu''', The Koch Institute at MIT

← Older revision		Revision as of 12:19, 14 July 2016
Line 51:		Line 51:
	With a variety of computing infrastructures available, building robust, transferable pipelines can increase utilization of compute resources. NGS analysis pipelines implemented as docker containers and deployed on a variety of compute platforms – (cluster, supercomputer, or workstation) will be discussed.		With a variety of computing infrastructures available, building robust, transferable pipelines can increase utilization of compute resources. NGS analysis pipelines implemented as docker containers and deployed on a variety of compute platforms – (cluster, supercomputer, or workstation) will be discussed.

		+	[[File:Sergi.pdf]]

	Speaker: '''Jingzhi Zhu''', The Koch Institute at MIT		Speaker: '''Jingzhi Zhu''', The Koch Institute at MIT

@@ Line 93: / Line 93: @@
   The evolving technology in big data, NGS, life science is really an evolution in what "big" means.  We've always dealt with challenging datasets but "big data" involves additional or more challenging work on the actual analysis and management processes--elaborate.
-The biggest problem is in the complexity of the projects.  But a larger problem is working with faculty who don't have a lot of money
+ The biggest problem is in the complexity of the projects.  But a larger problem is working with faculty who don't have a lot of money
 Most cores are willing to devote part of their time, pro bono, to generate results for grant submission.  The investigator will include the data and cost:effort into the grant for the analysis services.

@@ Line 79: / Line 79: @@
 Alberto Riva: The bioinformatics core facility sits within a closely organized group of core facilities that can be used for Life and Health Science.  Additionally, they have access to large and shared IT resources with segments setup for their specific use cases (a private area of the large 10,000 core cluster, for example).
-What fraction of the University of Florida system uses the core and how is work paid for?
+ What fraction of the University of Florida system uses the core and how is work paid for?
-        It's fee for service, but moving to a model where work is charged/allocated by level of effort for a resource with longer term projects through full resource allocation to a grant.
+The core is currently fee for service, but moving to a model where work is charged/allocated by level of effort for a resource with longer term projects through full resource allocation to a grant.
-	Regarding percentage of university using the core, there is no good measure.  Not everyone knows about the core, they focus on some outreach, but overall it's hard to quantify.
+Regarding percentage of university using the core, there is no good measure.  Not everyone knows about the core, they focus on some outreach, but overall it's hard to quantify.
-Discussion of overall cost that includes data analysis and storage.  One view was that storage is getting cheaper, however the data itself is still a problem: the data growing faster than storage is getting cheaper.  HMS, for example, has hired a data manager who works solely with people to put their data in the appropriate places--cheap archive storage vs. more expensive on-line high-performance storage.
+ Discussion of overall cost that includes data analysis and storage.
 At Purdue, there is not a single big large set of data, but 1000's of small datasets.  Purdue core works with users who have varying levels of analytic and IT knowledge. They find that they have spend time working on datasets in order to adapt/format/clean them for analysis as well as understanding the experimental parameters.  Not everyone knows what goes on inside and behind the scenes of the core in performing this work.  They expect the work to be quick, but without prior involvement in developing the experiment, it takes days to get the dataset to a state where it can be run through their analysis!  Educating the students and educating their users about the data, dataset and the analysis is important.
@@ Line 91: / Line 92: @@
 Biggest challenges for Riva is in educating users on how to generate the data--you may have all the big data you want, but if the experiment is not designed properly, there's quite a lot of cruft.
-The evolving technology in big data, NGS, life science is really an evolution in what "big" means.  We've always dealt with challenging datasets but "big data" involves additional or more challenging work on the actual analysis and management processes--elaborate.
+ The evolving technology in big data, NGS, life science is really an evolution in what "big" means.  We've always dealt with challenging datasets but "big data" involves additional or more challenging work on the actual analysis and management processes--elaborate.
- The biggest problem is in the complexity of the projects.  But a larger problem is working with faculty who don't have a lot of money
+The biggest problem is in the complexity of the projects.  But a larger problem is working with faculty who don't have a lot of money
- Most cores are willing to devote part of their time, pro bono, to generate results for grant submission.  The investigator will include the data and cost:effort into the grant for the analysis services.
+Most cores are willing to devote part of their time, pro bono, to generate results for grant submission.  The investigator will include the data and cost:effort into the grant for the analysis services.
-How do you deal with privacy and security of the data?  When thinking about a pipeline, do you take into acount what's public vs. private?
+ How do you deal with privacy and security of the data?  When thinking about a pipeline, do you take into acount what's public vs. private?
-  Purdue: download all data into their local environment.
+Purdue: download all data into their local environment.
-  Florida: they have the largest southern florida health center who works with patient data.  To comply with regulations, the research computing group has created a  secure area for their cluster to work with this data.  It's walled off from external and internal access--i.e controlled access.
+Florida: they have the largest southern florida health center who works with patient data.  To comply with regulations, the research computing group has created a  secure area for their cluster to work with this data.  It's walled off from external and internal access--i.e controlled access.
 Bottom line, last thought: a core and the personnel within it has to be adaptable in order to understand what is brought to them.  No 2 experiments are alike and needs continuously change.  The trends, technology capability and tools change.  Need to remain flexible, adapt pipelines, process and people.