Transcription
Scheduling PolicyJohn (TJ) KnoellerCondor Week 2017
Overview› Policy options in the SCHEDD Limits Job policy Mutating jobs Preventing changes2
Limits›››››››Max jobs runningMax jobs per submissionMax jobs per Owner (8.6)Max running DAGs per Owner (8.6)Max materialized jobs per cluster (8.7.1)Max active input transfersMax active output transfers3
User vs Owner vs Submitter› Owner attribute of job is OS ‘user’ Shadow impersonates Owner for file i/o Set by SCHEDD based on submit identity Immutable› Accounting ‘user’ a.k.a. Submitter Who’s quota/priority is checked/docked (Owner Nice) Domain AccountingGroup User can change at will4
Most limits are Submitter limits› “Fair” share is by submitter Negotiator only knows about submitters Priority / Quota Transfer queue› A few per-owner limits Max jobs per owner (8.6) Max running DAGs per owner (8.6)5
Monitoring the limits› Several talks on this on Thursday› Schedd Stats condor status –schedd –direct -long› Per submitter stats condor status –submit –long condor sos condor q –tot –long› Show jobs doing file transfer condor sos condor q –io6
Job policy› You want to have a policy about what jobsare allowed, or require certain attributes? Submit requirements Submit attributes Job transforms7
Example job policy› All jobs must have "Experiment" attribute Reject jobs that don't conform to the policySUBMIT REQUIREMENT NAMES (SUBMIT REQUIREMENT NAMES) CheckExpSUBMIT REQUIREMENT CheckExp \JobUniverse 7 Experiment isnt undefinedSUBMIT REQUIREMENT CheckExp REASON \"submissions must have Experiment"# JobUniverse 7 is Scheduler universe, i.e. DAGMAN.# JobUniverse 12 is Local universe, maybe except this also?8
Defaulting job attributes› Configure SUBMIT ATTRS to addattributes to jobs.SUBMIT ATTRS (SUBMIT ATTRS) ExperimentExperiment "CHTC"› Job ad starts with Experiment "CHTC"before the submit file is processed9
SUBMIT ATTRS›››››Good for setting defaultsWork happens outside of the SCHEDDUser can override or un-configureUnconditionalMay not happen with remote submit(Depends on who owns the config)10
Mutating jobs using jobtransforms (new in 8.6)› Configure JOB TRANSFORM *JOB TRANSFORM NAMES (JOB TRANSFORM NAMES) SetExpJOB TRANSFORM SetExp [ set Experiment "CHTC"; ]› Experiment "CHTC" written into eachjob ad as it is submitted.probably not a good thing in this case11
Transforming only some jobsJOB TRANSFORM NAMES (JOB TRANSFORM NAMES) SetExpJOB TRANSFORM SetExp @ end[Requirements JobUniverse ! 7 && Experiment is undefinedset Experiment "CHTC";]@end› Adds Experiment "CHTC" to each jobthat doesn't already have that attribute12
About job transforms› Converted to native syntax on startup› Job router syntax is loosely ordered copy delete set eval set› Native syntax is Confusing (and might be changing) Top to bottom Has temporary variables Has Conditionals13
Job transform native syntax# Use job transform to add pool constraint to vanilla jobs# based on whether the job needs GPUs or not#JOB TRANSFORM GPUS @ endREQUIREMENTS JobUniverse 5tmp.NeedsGpus (MY.RequestGPUs:0) 0if INT(tmp.NeedsGpus)SET Requirements (MY.Requirements) && (Pool "ICECUBE")elseSET Requirements (MY.Requirements) && (Pool "CHTC")endif@end14
Preventing change› IMMUTABLE JOB ATTRS Cannot be changed once set› PROTECTED JOB ATTRS Cannot be changed by the user› SECURE JOB ATTRS Like protected, but have security implicationsIMMUTABLE JOB ATTRS (IMMUTABLE JOB ATTRS) Experiment15
The motivating case for all this› How do I assign jobs to accounting groupsautomatically, while preventing users fromcheating? Job transforms Immutable attributes› But doing this in classad language is painfuleval set AcctGroup \IfThenElse(Owner "Bob","CHTC",IfThenElse(Owner "Alice","Math",IfThenElse(Owner "Al","Physics","Unknown")))16
Introducing Map files› Map file is text, with 3 fields per line› * key or regex result list ****BobAlice/.*Hat/i/.*/CHTC, SecurityCHTC, Math, PhysicsProblemCHTC› Yes, the first field must be *17
Defining a mapSCHEDD CLASSAD USER MAP NAMES MyMapCLASSAD USER MAPFILE MyMap /path/to/mapfile or SCHEDD CLASSAD USER MAPDATA MyMap @ end* Bob CHTC,Security* Alice CHTC,Math,Physics* /.*Hat/i Problem* /.*/ CHTC@endCan now use the userMap("MyMap") function in Classad expressions inthe SCHEDD.18
The Classad userMap functionresult userMap(mname, input) map input to first resultresult userMap(mname, input, preferred) map input to preferred resultresult userMap(mname, input, pref, def) map input to preferred or default result19
Putting it all togetherSCHEDD CLASSAD USER MAP NAMES (SCHEDD CLASSAD USER MAP NAMES) GroupsCLASSAD USER MAPFILE Groups /path/to/mapfile# Assign groups automaticallyJOB TRANSFORM NAMES AssignGroupJOB TRANSFORM AssignGroup @ end[copy Owner "AcctGroupUser";copy AcctGroup "RequestedAcctGroup";eval set AcctGroup nd# Prevent CheatingIMMUTABLE JOB ATTRS (IMMUTABLE JOB ATTRS) AcctGroup AcctGroupUserSUBMIT REQUIREMENT NAMES (SUBMIT REQUIREMENT NAMES) CheckGroupSUBMIT REQUIREMENT CheckGroup AcctGroup isnt undefinedSUBMIT REQUIREMENT CheckGroup REASON strcat("Could not map '", Owner, "' to a group")20
Or, to put it another wayuse FEATURE:AssignAccountingGroup(/path/map)You can runcondor config val use feature:AssignAccountingGroupto see what this metaknob expands to21
Any Questions?22
›Owner attribute of job is OS 'user' Shadow impersonates Owner for file i/o Set by SCHEDD based on submit identity Immutable ›Accounting 'user' a.k.a. Submitter Who'squota/priority is checked/docked (Owner Nice) Domain AccountingGroup User can change at will User vs Owner vs Submitter 4 ›"Fair"share is by submitter
