PROVE as a Web Service

Application development on EGEE with P-GRADE Portal Gergely Sipos MTA SZTAKI [email protected] EGEE Training and Induction EGEE Application Porting Support 1 Running jobs in EGEE from command line User Interface Write JDL, Submit workload

Workload Management System EGEE VO Retrieve status & (small) output files Workload: Single job One executable & one or more files Sequential or MPI Workflow DAG of jobs

Parameter study Independent, similar jobs 2 Defining a single job in JDL $ vi OR nano OR my.jdl JobType = "Normal"; Executable = gridTest; StdError = stderr.log; StdOutput = stdout.log; InputSandbox = {/home/sipos/gridTest}; OutputSandbox = {result.txt, stderr.log, stdout.log}; $ glite-wms-job-submit my.jdl JobID $ glite-wms-job-status JobID $ glite-wms-job-output JobID

3 Managing data in EGEE from command line User Interface Run file management commands Upload, transfer, register files EGEE VO Download files Delete files File catalog & Storage Elements $ lfc-mkdir myCatalogEntry

$ lfc-rmdir myCatalogEntry $ lcg-cr vo voName d SEName l myCatalogEntry file . . . 6 Short History of P-GRADE portal Parallel Grid Application Development Environment Initial development started in the Hungarian SuperComputing Grid project in 2003 Motivation: Simplify grid usage for application developers and end users It has been continuously developed since 2003 Around 30 manyear development + training + user support

Detailed information: Open Source community since January 2008: Current version: 2.8 7 Portal installations P-GRADE Portal services: Several VOs of EGEE: Biomed, Astronomy, Central European, NA4,... GILDA: Training VO of EGEE Southern Eastern European Grid Several national Grids (UK National Grid Service, HunGrid, Turkish Grid, etc.)

US Open Science Grid, TeraGrid OGF Grid Interoperability Now (GIN) VO Under development: KnowledgeGrid of Malaysia MIMOS & SZTAKI Portal services and account request: Account request form on portal login page 9 Design principles of P-GRADE portal P-GRADE Portal is not only a user interface, it is a General purpose Application Development and Execution Environment

P-GRADE Portal includes a high-level middleware layer for orchestrating jobs on grid resources inside a grid among several different grids and several VOs P-GRADE Portal is grid-neutral: It is not tailored to any particular middleware grid type Can be connected to various grids based on different grid middleware LCG-2, gLite, GT2, GT4, ARC, Unicore, etc. Implementation for GT2, GT4, gLite, ARC, LSF GUI is the same no matter which type of grid is connected to it 10 What is a P-GRADE Portal workflow?

A directed acyclic graph where Nodes represent jobs (batch programs to be executed on a computing element) Ports represent input/output files the jobs expect/produce Arcs represent file transfer operations semantics of the workflow: A job can be executed if all of its input files are available 11

Three levels of parallelism Multiple instances of the same workflow process different data files Job level: Parallel execution inside a workflow node (MPI job as workflow component) Workflow level: Parallel execution among workflow nodes (WF branch parallelism) PS workflow level: Parameter study execution of the workflow

Multiple jobs run parallel Each job can be a parallel program 12 Example: Computational Chemistry ~100 independent jobs to run Department of Chemistry, University of

Perugia SOLUTION OF SCHRODINGER EQUATION FOR TRIATOMIC SYSTEMS USING TIMEDEPENDENT (RWAVEPR) OR TIME INDEPENDENT (ABC) METHOD A single execution can be between 5 hours and 10 hours Many simulations at the same time SEQUENTIAL FORTRAN 90 13 Typical user scenario Job compilation phase UPLOAD JOB SOURCE(S)

Portal server Client COMPILE EDIT Grid services DOWNLOAD BINARI(ES) 14 Typical user scenario Workflow development phase


BINARIES DSpace WF repository 15 Typical user scenarios Workflow execution phase MyProxy Certificate servers TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES


(SMALL) RESULTS DOWNLOAD (SMALL) RESULTS 16 P-GRADE Portal structural overview Source: Jia Yu and Rajkumar Buyya: A Taxonomy of Workflow Management Systems for Grid Computing, Journal of Grid Computing, Volume 3, Numbers 3-4 / September, 2005 Web browser DSpace workflow repository

Java Webstart workflow editor Workflow language: Extended DAGMan Globus and gLite information systems (Globus GIIS, gLite BDII) Workflow scheduler: Extended DAGMan Grid interface: command line clients + scripts EGEE, Globus (and ARC) Grid services

(gLite WMS, LFC,; Globus GRAM, ) 18 Web interface - Portlets 19 Email notifications NOTIFY 20 Workflow portlet WORKFLOW EDITOR

21 Graphical workflow editing To define a graph: 1. Drag & drop components: jobs and ports 2. Define their properties 3. Connect ports by channels (no cycles, no loops) System generates JDL for each job automatically 24

Workflow Editor Properties of a job Properties of a job: Executable file Type of executable (Sequential / Parallel) Command line parameters Which resource to use? Which VO? Broker or Computing element? 25 Workflow Editor

Defining input-output files File properties Type: input: the executable reads output: the executable generates File type: local: comes from my desktop remote: comes from an SE File: location of the file Internal file name: Executable uses this e.g. fopen(, ) File storage type (output files only): Permanent: final result Volatile: temp. data channel

26 Scaling up a workflow to a parameter study Complete workflow Grid files (files in the same EGEE catalog entry) Results stored in grid files 27 Advanced parameter studies Generator

component(s) Initial input data Generate or cut input into smaller pieces Complete workflow Collector component(s) Aggregate

result 28 Concept of parameter study workflows GEN Generator part generates the input parameter space SEQ SEQ SEQ SEQ

Parameter study part COLL Collector part evaluates and integrates the results 29 Example: Image processing /grid/gilda/sipos/InputImages Image.0 Image.1 Image.2 /grid/gilda/sipos/XCoordinates

XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 3 x 2 x 2 = 12 execution of the whole workflow CROSS PRODUCT of data items /grid/gilda/sipos/Output ImagePart.0 ImagePart.1 ...

30 Upload a workflow from client side or from FTP server UPLOAD STORED on FTP server 32 Import a workflow from DSpace repository 34 Proxy creation:

The first step of grid usage Workload Management Write JDL, Submit job (executable + small inputs)System (WMS) User Interface (UI) or Portal Information System (BDII) query create proxy Retrieve status & (small) output files

Retrieve output Job status publish state Submit job Loggin g Site X of YOUR VO Computing Element (CE)

Job status VO Management Service (VOMS) Logging and bookkeeping (LB) process Storage Element (SE) Read/write files

36 Certificate and proxy management Portlet 37 OGF GIN interoperability portal by P-GRADE Acccessing Globus, gLite and ARC based grids/VOs simultaneously Proxy 1 P-GRADE Proxy 6 P-GRADE GEMLCA portal

Portal Proxy 2 Proxy 5 GEMLCA Repository Proxy 3 Proxy 4 38 Application execution 39

Detailed view of a PS workflow Generator job(s) Overall statistics of workflow instances Workflow instances Collector job(s) 40 Fault-tolerant execution Utilizing Condor DAGMans rescue mechanism Resubmit only failed jobs, not completed jobs! If the EGEE broker leaves a job stuck in a CEs

queue, the portal automatically kills the job on this site and resubmits the job to the broker by prohibiting this site. As a result the portal guarantees the correct submission of a job as long as there exists at least one matching resource job submission is reliable even in an unreliable grid 41 Information system visualization 42 LFC-SE file browser portlet

43 Compilation support 44 Learn once, use everywhere Develop once, execute anywhere Thank you! [email protected] 45

Recently Viewed Presentations

  • Presentación de PowerPoint

    Presentación de PowerPoint

    Henry Cavendish (1731-1810) En 1766, experimentando con ácidos fuertes y metales como el zinc, obtuvo por primera vez hidrógeno, al que llamó flogisto o 'aire inflamable', y descubrió que es el más ligero de los gases. Montgolfier (1783) Carl Wilhelm...
  • Team Building -

    Team Building -

    Team Building Sharon Doner-Feldman Training for You What is Teamwork & Team Building Teamwork Concept of people working together as a team Team player A team player is someone who is able to get along with their colleagues and work...
  • CHAPTER 7 Natural Environments of North America Section

    CHAPTER 7 Natural Environments of North America Section

    SECTION 1 Physical Features Question: What are the various landform regions of the United States and Canada? Region Location Description SECTION 1 Physical Features Gulf-Atlantic Coastal Plain Piedmont Appalachian Highlands along the Atlantic Ocean and Gulf of Mexico Alabama to...
  • Introduction to infant and young child feeding

    Introduction to infant and young child feeding

    explain the use of processed complementary foods ... Gap for iron Key Message 4 Animal-source foods are especially good for children, to help them grow strong and lively Key Message 4 Animal-source foods are especially good for children, to help...
  • Americas Children Need a Rescue Operation by Vera

    Americas Children Need a Rescue Operation by Vera

    Each diagnosis---a different regimen of drugs. At age 11 "bipolar disorder" — a whole new set of Rx. New York Times 11/ 2006 DSM-IV: Pathology Grab Bag "Bipolar Child" Historically rare, now "highly prevalent" "Five years ago, pinning down bipolar...
  • TRA 5514B : Terminologie transsystmique et documentation :

    TRA 5514B : Terminologie transsystmique et documentation :

    Canadian Abridgment : un petit rappel N'oubliez pas de toujours consulter le supplément pour chaque série pour vous assurer que vous avez les informations les plus à jour. Les suppléments se retrouvent à deux endroits : à côté de chaque...
  • Tok Essay

    Tok Essay

    TOK ESSAY The title sets the scope of the question. Your job is to find the knowledge issues within that scope. Then, decide what AoK and WoK to include:
  • Présentation PowerPoint - BIOMACH

    Présentation PowerPoint - BIOMACH

    A "brownian" rotary motor based on decacyclene pp.531-533 Hexa-t-Butyl-Decacyclene HB-DC Top (A) and side (B) views of models of the molecular structure of hexa-tert-butyl decacyclene (HB-DC). The molecule consists of a central conjugated decacyclene core with six t-butyl legs attached...