Issue212

Title unify gkigrid and local jobs
Priority wish Status resolved
Superseder Nosy List gabi, jendrik, malte
Assigned To jendrik Keywords
Optional summary

Created on 2011-01-22.17:41:48 by malte, last changed by jendrik.

Messages
msg2408 (view) Author: jendrik Date: 2013-03-22.17:45:20
Closing this as discussed offline.
msg2403 (view) Author: malte Date: 2013-03-22.15:32:39
I'm not sure we would need a wrapper script or if clauses or that we can't
directly support local experiments on multiple cores. Maybe we can discuss this
in person.
msg2402 (view) Author: jendrik Date: 2013-03-22.15:02:52
I would like to leave the two scripts separate, because
- The approach has been extensively used and tested by now
- The grid bash script would need a wrapper script that repeatedly calls the grid script with 
increasing SGE_TASK_IDs. This and a local file with 10.000 if clauses is rather ugly.
- The current approach supports running local experiments on multiple cores
- A change can always be made when need arises

Gabi and Florian agree, what do you think Malte?
msg1224 (view) Author: gabi Date: 2011-01-24.18:51:30
One argument for the one-file solution is that it increases my confidence that
the local experiments do the same thing, while two scripts often tend to diverge.
msg1218 (view) Author: jendrik Date: 2011-01-22.21:39:09
Another advantage of the two-files solution is the possibility to run local 
experiments in parallel with the multiprocessing package. Further all the 
customizations for local jobs might be easier in python (at least for me ;)
msg1217 (view) Author: malte Date: 2011-01-22.20:05:40
I thought that since grid jobs are just shell scripts with some special comments
at the top, it'd be nice to have everything in one place (I think it might also
simplify the experiment code if it didn't have to generate two very similar but
subtly different kinds of files), and I don't think it'd be difficult to provide
different kinds of customization if necessary. But of course creating two files
is also a good possibility. Gabi, any preferences?
msg1216 (view) Author: jendrik Date: 2011-01-22.20:00:03
Why not just create both executables, local-run and exp.q in the 
experiment directory? Their behaviour should be consistent since they 
just change into each run's directory and call the run command. Having 
two different executables would also allow for easier customization.
msg1213 (view) Author: malte Date: 2011-01-22.17:41:48
Currently, I usually first generate a job as a local job to be able to test it,
and then generate it as a gkigrid job when I'm convinced that it works. It would
be nice if there were a way to generate a job in such a way that it can be used
in both modes.

For example, the *.q files could detect whether they are called by the grid
(e.g. via the environment variables that also tell it the job id) and behave in
a special way in that case, and act as regular shell scripts otherwise.

There are a few more enhancements that would be useful in this context, e.g.
more verbose output when run as local jobs, or a way to clean up and restore a
job to its original state after a local test run. Maybe we can discuss this
further at one of our next meetings.
History
Date User Action Args
2013-03-22 17:45:20jendriksetstatus: chatting -> resolved
messages: + msg2408
2013-03-22 15:32:39maltesetmessages: + msg2403
2013-03-22 15:02:52jendriksetmessages: + msg2402
2011-01-24 18:51:31gabisetmessages: + msg1224
2011-01-23 18:39:41jendriksetassignedto: jendrik
2011-01-22 21:39:09jendriksetmessages: + msg1218
2011-01-22 20:05:40maltesetmessages: + msg1217
2011-01-22 20:00:03jendriksetmessages: + msg1216
2011-01-22 17:41:48maltecreate