CrocoBLAST:Job management - Revision history

Crina at 05:26, 25 July 2016

2016-07-25T05:26:59Z

← Older revision		Revision as of 05:26, 25 July 2016
Line 21:		Line 21:
	</div>		</div>
	</div>		</div>


	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	=Manage databases=		=Manage databases=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
Line 79:		Line 79:
	</div>		</div>
	</div>		</div>


	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	=Manage CrocoBLAST queue=		=Manage CrocoBLAST queue=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">

Crina: /* Create BLAST jobs */

2016-07-25T05:26:06Z

Create BLAST jobs

← Older revision		Revision as of 05:26, 25 July 2016
Line 23:		Line 23:

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	=Manage databases=		=Manage databases=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">

Crina: /* Add database from your computer */

2016-07-25T05:25:49Z

Add database from your computer

← Older revision		Revision as of 05:25, 25 July 2016
Line 59:		Line 59:
	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	===Add database from your computer===		==Add database from your computer==
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	If you have already downloaded the databases from NCBI, or if you do not have internet connection, you may add to the CrocoBLAST index database files stored on your computer. Remember to provide a unique and representative name for each database you add, so that it is easy to call the databases later. If the database files are appropriately formatted (e.g., psq or nsq):		If you have already downloaded the databases from NCBI, or if you do not have internet connection, you may add to the CrocoBLAST index database files stored on your computer. Remember to provide a unique and representative name for each database you add, so that it is easy to call the databases later. If the database files are appropriately formatted (e.g., psq or nsq):
Line 80:		Line 80:

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	=Manage CrocoBLAST queue=		=Manage CrocoBLAST queue=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">

Crina: /* Retrieve database from the NCBI servers */

2016-07-25T05:25:39Z

Retrieve database from the NCBI servers

← Older revision		Revision as of 05:25, 25 July 2016
Line 44:		Line 44:

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">
	===Retrieve database from the NCBI servers===		==Retrieve database from the NCBI servers==
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	In the most typical scenario, you will use the [ftp://ftp.ncbi.nlm.nih.gov/refseq/release/ established reference sequence databases maintained by NCBI]. CrocoBLAST allows you to specify the name of such a database, and will download or update the database for you:		In the most typical scenario, you will use the [ftp://ftp.ncbi.nlm.nih.gov/refseq/release/ established reference sequence databases maintained by NCBI]. CrocoBLAST allows you to specify the name of such a database, and will download or update the database for you:
Line 58:		Line 58:

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">

	===Add database from your computer===		===Add database from your computer===
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">

Crina: /* Add database from your computer */

2016-07-25T05:24:28Z

Add database from your computer

← Older revision		Revision as of 05:24, 25 July 2016
Line 78:		Line 78:
	</div>		</div>

			<div class="mw-collapsible mw-collapsed">
	~~#==~~=Manage CrocoBLAST queue===		=Manage CrocoBLAST queue=
			<div class="mw-collapsible-content">
	The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.		The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.
			</div>
			</div>

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">
	~~##==~~==Execution====		==Execution==
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	Say you have ''created one or more BLAST jobs'' and are ready to start munching some sequences. It's easy:		Say you have ''created one or more BLAST jobs'' and are ready to start munching some sequences. It's easy:
Line 125:		Line 127:

	This will provide you with information regarding which jobs are queued, with full details regarding the job ID and BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results.		This will provide you with information regarding which jobs are queued, with full details regarding the job ID and BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results.
			</div>
			</div>

	~~##=~~===Administration====		<div class="mw-collapsible mw-collapsed">
			==Administration==
			<div class="mw-collapsible-content">
	If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run. Subsequently, you may perform operations like adding, removing, or reordering jobs in the queue:		If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run. Subsequently, you may perform operations like adding, removing, or reordering jobs in the queue:

Line 147:		Line 153:

	<div class="mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">
	~~#==~~=Generate report===		=Generate report=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
	CrocoBLAST logs all your activity, so that you may always review your work. To output this information:		CrocoBLAST logs all your activity, so that you may always review your work. To output this information:

Crina at 05:22, 25 July 2016

2016-07-25T05:22:10Z

← Older revision		Revision as of 05:22, 25 July 2016
Line 1:		Line 1:
			__NOTOC__
	CrocoBLAST is built to help you plan your BLAST jobs and run them efficiently. CrocoBLAST operates with the concept of queue, which is basically a list of BLAST jobs scheduled to run. Thus, you can plan several BLAST job and let CrocoBLAST manage their execution for you.		CrocoBLAST is built to help you plan your BLAST jobs and run them efficiently. CrocoBLAST operates with the concept of queue, which is basically a list of BLAST jobs scheduled to run. Thus, you can plan several BLAST job and let CrocoBLAST manage their execution for you.

Line 7:		Line 8:


	<div class="~~toccolours~~ mw-collapsible mw-collapsed">		<div class="mw-collapsible mw-collapsed">
	=Create BLAST jobs=		=Create BLAST jobs=
	<div class="mw-collapsible-content">		<div class="mw-collapsible-content">
Line 21:		Line 22:
	</div>		</div>

			<div class="mw-collapsible mw-collapsed">
	=Manage databases=		=Manage databases=
			<div class="mw-collapsible-content">
	To submit a BLAST job, you must specify which database you wish to align against. The first time you indicate a database for a BLAST job, CrocoBLAST will remember it and add it to its index, so that in the future it is easier for you to access this database. You can see which databases are already indexed in CrocoBLAST:		To submit a BLAST job, you must specify which database you wish to align against. The first time you indicate a database for a BLAST job, CrocoBLAST will remember it and add it to its index, so that in the future it is easier for you to access this database. You can see which databases are already indexed in CrocoBLAST:

Line 38:		Line 40:

	There are two ways to add a new database to the CrocoBLAST index. In both cases, you should provide a simple name for new each database, so that you may later refer this database easily whenever you need to run a BLAST job.		There are two ways to add a new database to the CrocoBLAST index. In both cases, you should provide a simple name for new each database, so that you may later refer this database easily whenever you need to run a BLAST job.
			</div>
			</div>

	==Retrieve database from NCBI servers==		<div class="mw-collapsible mw-collapsed">
			===Retrieve database from the NCBI servers===
			<div class="mw-collapsible-content">
	In the most typical scenario, you will use the [ftp://ftp.ncbi.nlm.nih.gov/refseq/release/ established reference sequence databases maintained by NCBI]. CrocoBLAST allows you to specify the name of such a database, and will download or update the database for you:		In the most typical scenario, you will use the [ftp://ftp.ncbi.nlm.nih.gov/refseq/release/ established reference sequence databases maintained by NCBI]. CrocoBLAST allows you to specify the name of such a database, and will download or update the database for you:

Line 48:		Line 54:

	When adding or updating a database in this manner, you need not worry about the format of the database, as NCBI provides pre-formatted database files.		When adding or updating a database in this manner, you need not worry about the format of the database, as NCBI provides pre-formatted database files.
			</div>
			</div>

	==Add database from your computer==		<div class="mw-collapsible mw-collapsed">
			===Add database from your computer===
			<div class="mw-collapsible-content">
	If you have already downloaded the databases from NCBI, or if you do not have internet connection, you may add to the CrocoBLAST index database files stored on your computer. Remember to provide a unique and representative name for each database you add, so that it is easy to call the databases later. If the database files are appropriately formatted (e.g., psq or nsq):		If you have already downloaded the databases from NCBI, or if you do not have internet connection, you may add to the CrocoBLAST index database files stored on your computer. Remember to provide a unique and representative name for each database you add, so that it is easy to call the databases later. If the database files are appropriately formatted (e.g., psq or nsq):

Line 65:		Line 75:
	CrocoBLAST -add_database --sequence_file '''protein''' <span style="color:green">fastq_file</span> <span style="color:orange">database_name</span> <span style="color:green">output_folder</span>		CrocoBLAST -add_database --sequence_file '''protein''' <span style="color:green">fastq_file</span> <span style="color:orange">database_name</span> <span style="color:green">output_folder</span>
	</code>		</code>
			</div>
			</div>


	=Manage CrocoBLAST queue=		#===Manage CrocoBLAST queue===

	The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.		The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.

	==Execution==		<div class="mw-collapsible mw-collapsed">
			##====Execution====
			<div class="mw-collapsible-content">
	Say you have ''created one or more BLAST jobs'' and are ready to start munching some sequences. It's easy:		Say you have ''created one or more BLAST jobs'' and are ready to start munching some sequences. It's easy:

Line 112:		Line 126:
	This will provide you with information regarding which jobs are queued, with full details regarding the job ID and BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results.		This will provide you with information regarding which jobs are queued, with full details regarding the job ID and BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results.

	==Administration==		##====Administration====
	If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run. Subsequently, you may perform operations like adding, removing, or reordering jobs in the queue:		If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run. Subsequently, you may perform operations like adding, removing, or reordering jobs in the queue:

Line 128:		Line 142:
	CrocoBLAST -status		CrocoBLAST -status
	</code>		</code>
			</div>
			</div>


			<div class="mw-collapsible mw-collapsed">
	=Generate report=		#===Generate report===
			<div class="mw-collapsible-content">
	CrocoBLAST logs all your activity, so that you may always review your work. To output this information:		CrocoBLAST logs all your activity, so that you may always review your work. To output this information:

Line 137:		Line 154:
	CrocoBLAST -generate_report		CrocoBLAST -generate_report
	</code>		</code>
			</div>
			</div>

	'''Return to the [[CrocoBLAST:UserManual \| Table of contents]]'''		'''Return to the [[CrocoBLAST:UserManual \| Table of contents]]'''

Crina: /* Create BLAST jobs */

2016-07-25T04:33:32Z

Create BLAST jobs

← Older revision		Revision as of 04:33, 25 July 2016
Line 7:		Line 7:


			<div class="toccolours mw-collapsible mw-collapsed">
	=Create BLAST jobs=		=Create BLAST jobs=
			<div class="mw-collapsible-content">
	As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.		As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.

Line 17:		Line 18:

	Note that, when you create a BLAST job, CrocoBLAST automatically assigns each BLAST job a unique job ID, and updates the CrocoBLAST queue (more on this later).		Note that, when you create a BLAST job, CrocoBLAST automatically assigns each BLAST job a unique job ID, and updates the CrocoBLAST queue (more on this later).
			</div>
			</div>

	=Manage databases=		=Manage databases=

Crina at 04:18, 25 July 2016

2016-07-25T04:18:17Z

@@ Line 2: / Line 2: @@
 All CrocoBLAST functionality is available via the command line utility and the graphical user interface. In fact, the graphical user interface does precisely what its name suggests: it provides an interface for the command line utility. In a nutshell, while you can interact with CrocoBLAST via simple commands, you may also use the interface to generate the commands or read the output of such commands.
@@ Line 123: / Line 126: @@
 CrocoBLAST -status
 </code>

Crina at 04:13, 25 July 2016

2016-07-25T04:13:12Z

← Older revision		Revision as of 04:13, 25 July 2016
Line 5:		Line 5:

	=Create BLAST jobs=		=Create BLAST jobs=

	As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.		As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.

Line 12:		Line 13:
	</code>		</code>

	Note that ~~creating~~ a BLAST job automatically updates the CrocoBLAST queue (more on this later).		Note that, when you create a BLAST job, CrocoBLAST automatically assigns each BLAST job a unique job ID, and updates the CrocoBLAST queue (more on this later).


	=Manage databases=		=Manage databases=

	To submit a BLAST job, you must specify which database you wish to align against. The first time you indicate a database for a BLAST job, CrocoBLAST will remember it and add it to its index, so that in the future it is easier for you to access this database. You can see which databases are already indexed in CrocoBLAST:		To submit a BLAST job, you must specify which database you wish to align against. The first time you indicate a database for a BLAST job, CrocoBLAST will remember it and add it to its index, so that in the future it is easier for you to access this database. You can see which databases are already indexed in CrocoBLAST:

Line 21:		Line 24:
	</code>		</code>

	~~You can~~ provide a simple name for each database, that you may later refer to whenever you need to run a BLAST job~~. There are two ways to add a new database to the CrocoBLAST index~~.		If you want to remove a database from the CrocoBLAST index (for example, because it has become obsolete), you need to first specify the type of '''sequences''' it holds, and, of course, the name of the database.

			<code>
			CrocoBLAST -remove_database '''nucleotide''' <span style="color:blue">database_name</span><br>
			CrocoBLAST -remove_database '''protein''' <span style="color:blue">database_name</span>
			</code>


			There are two ways to add a new database to the CrocoBLAST index. In both cases, you should provide a simple name for new each database, so that you may later refer this database easily whenever you need to run a BLAST job.

	==Retrieve database from NCBI servers==		==Retrieve database from NCBI servers==
Line 49:		Line 60:
	CrocoBLAST -add_database --sequence_file '''protein''' <span style="color:green">fastq_file</span> <span style="color:orange">database_name</span> <span style="color:green">output_folder</span>		CrocoBLAST -add_database --sequence_file '''protein''' <span style="color:green">fastq_file</span> <span style="color:orange">database_name</span> <span style="color:green">output_folder</span>
	</code>		</code>


	=Manage CrocoBLAST queue=		=Manage CrocoBLAST queue=

	The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.		The efficiency of CrocoBLAST lies in its ability to parallelize the execution of your BLAST jobs. This is related to breaking each big calculation into smaller pieces, and then organizing the execution of the pieces. Having smaller pieces means that you need less memory to run each job, and if you can analyze several pieces at once you can speed up the total calculation time. CrocoBLAST takes care of these things for you.

Line 92:		Line 105:
	</code>		</code>

	This will provide you with information regarding which jobs are queued, with full details regarding the BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results. If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run.		This will provide you with information regarding which jobs are queued, with full details regarding the job ID and BLAST setup, as well as a description about the progress of the alignment. The progress of each job is described in three main directions: fragmentation of the input file, alignment, and assembly of results.

			==Administration==
			If you want to change anything about the queue (say, pause one job and start another, or change the order of the jobs in a queue), you need to first pause or stop the current run. Subsequently, you may perform operations like adding, removing, or reordering jobs in the queue:

			<code>
			CrocoBLAST -add_to_queue <span style="color:blue">blast_program database</span> <span style="color:green">input_file output_folder</span><br>
			CrocoBLAST -remove_from_queue <span style="color:blue">job_id</span><br>
			CrocoBLAST -remove_from_queue <span style="color:blue">job_id_1 job_id_2 ...</span><br>
			CrocoBLAST -move_top_queue <span style="color:blue">job_id</span><br>
			CrocoBLAST -move_top_queue <span style="color:blue">job_id_pos_1 job_id_pos_2 ...</span>
			</code>

			Note that, once a job is added to the queue, you may perform operations with it (remove, reorder) if you refer to the job by its job ID. You can obtain the job IDs by checking the current state of the queue:

			<code>
			CrocoBLAST -status
			</code>

Crina: /* Create BLAST job */

2016-07-25T03:58:22Z

Create BLAST job

← Older revision		Revision as of 03:58, 25 July 2016
Line 4:		Line 4:


	=Create BLAST ~~job~~=		=Create BLAST jobs=
	As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.		As already mentioned, BLAST takes an input file with unknown sequences and aligns each such sequence against a database of known sequences. To create a job, you must first specify the [http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml BLAST program] you plan to use, which depends on the nature of the unknown sequences in your input file, and the nature of the sequences in the reference database. Then, you need to specify the name of the ''database'' listed in the CrocoBLAST index (more details on that below) that contains the reference sequences you wish to use. Finally, provide the input file and the location where you want CrocoBLAST to place the output files. Keep in mind that the output files may be quite large. Finally, if you want to change the default BLAST settings, you can do so by specifying the [http://www.ncbi.nlm.nih.gov/books/NBK279675/ names and values of the BLAST options] of interest.

Line 12:		Line 12:
	</code>		</code>

	Note that creating a BLAST job automatically updates the CrocoBLAST queue (more on this later).		Note that creating a BLAST job automatically updates the CrocoBLAST queue (more on this later).



	=Manage databases=		=Manage databases=