...

Big Data - Apache Oozie

Back to Course

Lesson Description


Lession - #746 Apache Oozie Coordinator


Users generally run map- reduce, hadoop- streaming, hdfs and/ or Pig jobs on the grid. Multiple of these jobs can be combined to form a workflow job. Oozie, Hadoop Workflow System defines a workflow system that runs such jobs.

Generally, workflow jobs are run based on regular time intervals and/ or data availability. And, in some cases, they can be triggered by an external event.

Expressing the condition( s>
that trigger a workflow job can be modeled as a predicate that has to be satisfied. The workflow job is started after the predicate is satisfied. A predicate can reference to data, time and/ or external events. In the future, the model can be extended to support additional event types.

It's also necessary to connect workflow jobs that run regularly, but at different time intervals. The outputs of multiple subsequent runs of a workflow become the input to the next workflow. For illustration, the outputs of last 4 runs of a workflow that runs every 15 minutes come the input of another workflow that runs every 60 minutes. Chaining together these workflows result it's referred as a data application pipeline.

The Oozie Coordinator system allows the user to define and execute recurrent and interdependent workflow jobs( data application pipelines>
.

Real world data application pipelines have to account for reprocessing, late processing, catchup, partial processing, monitoring, notification and SLAs.

This document defines the functional specification for the Oozie Coordinator system.

Coordinator Job Status

Similar to the workflow, parameters can be passed to a coordinator also using the. properties file. These parameters are resolved using the configuration properties of Job configuration used to submit the coordinator job.

However, the value of the parameter will be undefined and the job submission will fail, If a configuration property used in the definitions isn't provided with the job configuration used to submit a coordinator job.

At any time, a coordinator job is in one of the following statuses − PREP, RUNNING, PREPSUSPENDED, SUSPENDED, PREPPAUSED, PAUSED, SUCCEEDED, DONWITHERROR, KILLED, FAILED.

Valid coordinator job status transitions are −


PREP − PREPSUSPENDED| PREPPAUSED| RUNNING| KILLED

RUNNING − SUSPENDED| PAUSED| SUCCEEDED| DONWITHERROR| KILLED| FAILED

PREPSUSPENDED − PREP| KILLED

SUSPENDED − RUNNING| KILLED

PREPPAUSED − PREP| KILLED

PAUSED − SUSPENDED| RUNNING| KILLED

When a coordinator job is submitted, Oozie parses the coordinator job XML. Oozie then creates a record for the coordinator with status PREP and returns a unique ID. The coordinator is also started immediately if the pause time isn't set.

When a user requests to suspend a coordinator job that's in status PREP, Oozie puts the job in the status PREPSUSPEND. Similarly, when the pause time reaches for a coordinator job with the status PREP, Oozie puts the job in the status PREPPAUSED.

Again, when a user requests to resume a PREPSUSPEND coordinator job, Oozie puts the job in status PREP. And when the pause time is reset for a coordinator job and job status is PREPPAUSED, Oozie puts the job in status PREP.

When a coordinator job starts, Oozie puts the job in status RUNNING and starts materializing workflow jobs based on the job frequency.

When a user requests to kill a coordinator job, Oozie puts the job in status KILLED and it sends kill to all submitted workflowjobs.However, Oozie puts the coordinator job into DONEWITHERROR, If any coordinator action finishes with not KILLED.

When a user requests to suspend a Coordinator job that's in status RUNNING, Oozie puts the job in status SUSPEND and it suspends all the submitted workflow jobs.

When pause time reaches for a Coordinator job that's in status handling, Oozie puts the job in status Broke.

Again, when a stoner requests to renew a SUSPEND Coordinator job, Oozie puts the job in status RUNNING. And when pause time is reset for a Coordinator job and job status is PAUSED, Oozie puts the job in status RUNNING.

A Coordinator job creates workflow jobs( generally Coordinator actions>
only for the duration of the Coordinator job and only if the Coordinator job is in RUNNINGstatus.However, when resumed it'll produce all the Coordinator actions that should have been created during the time it was suspended, conduct won't be lost, If the Coordinator job has been suspended.

When the Coordinator job materialization finishes and all the workflow jobs finish, Oozie updates the Coordinator status accordingly. For illustration, if all the workflows are SUCCEEDED, Oozie puts the Coordinator job into SUCCEEDED status. However, if any workflow job finishes with not SUCCEEDED(e.g. KILLED or FAILED or TIMEOUT>
, also Oozie puts the Coordinator job intoDONEWITHERROR.However, Oozie puts the Coordinator job into DONEWITHERROR, If all Coordinator actions are TIMEDOUT.

Reference − http//oozie.apache.org/docs/>


Parametrization of a Coordinator

The workflow parameters can be passed to a Coordinator as well using the. properties file. These parameters are resolved using the configuration properties of Job configuration used to submit the Coordinator job.

However, the value of the parameter will be undefined and the job submission will fail, If a configuration property used in the definition isn't provided with the job configuration used to submit a Coordinator job.