\section{Efficiency And Maintainability}
\label{Efficiency And Maintainability}

Efficiency (in the sense of {\em economy of suite definition}) and
maintainability go hand in hand. This section describes techniques for clean
and efficient construction of complex workflows that are easy to understand,
maintain, and modify.

\subsection{The Task Family Hierarchy}
\label{The Task Family Hierarchy}

A properly designed family hierarchy fulfills three purposes in Cylc:

\begin{itemize}
  \item efficient sharing of all configuration common to groups of related
    tasks
  \item efficient bulk triggering, for clear scheduling graphs
  \item clean suite visualization and monitoring, because families are
    collapsible in the GUIs
\end{itemize}

\subsubsection{Sharing By Inheritance}
\label{Sharing By Inheritance}

Duplication is a maintenance risk because changes have to be repeated in
multiple places without mistakes. On the other hand, unnecessary sharing of
items via global variables is also bad because it is hard to be sure which
tasks are using which variables. A properly designed runtime inheritance
hierarchy can give every task exactly what it needs, and nothing that it
doesn't need.

If a group of related tasks has some configuration in common, it can be
factored out into a task family inherited by all.

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[OBSPROC]]
        # Settings common to all obs processing tasks.
    [[obs1]]
        inherit = OBSPROC
    [[obs2]]
        inherit = OBSPROC
\end{lstlisting}

If several families have settings in common, they can in turn can inherit
from higher-level families. 

Multiple inheritance allows efficient sharing even for overlapping categories
of tasks. For example consider that some obs processing tasks in the following
suite run parallel jobs and some serial:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[SERIAL]]
        # Serial job settings.
    [[PARALLEL]]
        # Parallel job settings.
    [[OBSPROC]]
        # Settings for all obs processing tasks.
    [[obs1, obs2, obs3]]
        # Serial obs processing tasks.
        inherit = OBSPROC, SERIAL
    [[obs4, obs5]]
        # Parallel obs processing tasks.
        inherit = OBSPROC, PARALLEL
\end{lstlisting}

Note that suite parameters should really be used to define family members
efficiently - see Section~\ref{Generating Tasks}.

Cylc provides tools to help make sense of your inheritance hierarchy:

\begin{itemize}
  \item \lstinline=cylc graph -n/--namespaces= - plot the full multiple
      inheritance graph (not the dependency graph)
  \item \lstinline=cylc get-config SUITE= - print selected sections or items
      after inheritance processing
  \item \lstinline=cylc graph SUITE= - plot the dependency graph, with
      collapsible first-parent families (see~\ref{Task Families And Visualization})
  \item \lstinline=cylc list -t/--tree SUITE= - print the first-parent
    inheritance hierarchy
  \item \lstinline=cylc list -m/--mro SUITE= - print the inheritance
      precedence order for each runtime namespace
\end{itemize}

\subsubsection{Family Triggering}

Task families can be used to simplify the scheduling graph wherever many
tasks need to trigger at once:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = pre => MODELS
[runtime]
    [[MODELS]]
    [[model1, model2, model3, ...]]
        inherit = MODELS
\end{lstlisting}

To trigger {\em off of} many tasks at once, family names need to be qualified
by \lstinline@<state>-all@ or \lstinline@<state>-any@ to indicate the desired
member-triggering semantics:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """pre => MODELS
                MODELS:succeed-all => post"""
\end{lstlisting}

Note that this can be simplified further because Cylc ignores trigger
qualifiers like \lstinline=:succeed-all= on the right of trigger arrows
to allow chaining of dependencies:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = pre => MODELS:succeed-all => post
\end{lstlisting}

\subsubsection{Family-to-Family Triggering}

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = BIG_FAM_1:succeed-all => BIG_FAM_2
\end{lstlisting}

This means every member of \lstinline=BIG_FAM_2= depends on every member
of \lstinline=BIG_FAM_1= succeeding. For very large families this can create so
many dependencies that it affects the performance of Cylc at run time, as
well as cluttering graph visualizations with unnecessary edges. Instead,
interpose a dummy task that signifies completion of the first family:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = BIG_FAM_1:succeed-all => big_fam_1_done => BIG_FAM_2
\end{lstlisting}

For families with \lstinline=M= and \lstinline=N= members respectively, this 
reduces the number of dependencies from \lstinline=M*N= to \lstinline=M+N=
without affecting the scheduling.

\includegraphics[width=\textwidth]{resources/png/fam-to-fam-1.png}
\includegraphics[width=\textwidth]{resources/png/fam-to-fam-2.png}

\subsubsection{Task Families And Visualization}
\label{Task Families And Visualization}

{\em First parents} in the inheritance hierarchy double as collapsible summary
groups for visualization and monitoring. Tasks should generally be grouped into
visualization families that reflect their logical purpose in the suite rather
than technical detail such as inherited job submission or host settings. So in
the example under Section~\ref{Sharing By Inheritance} above all
\lstinline=obs<n>= tasks collapse into \lstinline=OBSPROC= but not into
\lstinline=SERIAL= or \lstinline=PARALLEL=.

If necessary you can introduce new namespaces just for visualization:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[MODEL]]
        # (No settings here - just for visualization).
    [[model1, model2]]
        inherit = MODEL, HOSTX
    [[model3, model4]]
        inherit = MODEL, HOSTY
\end{lstlisting}

To stop a solo parent being used in visualization, demote it to secondary with
a null parent like this:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[SERIAL]]
    [[foo]]
        # Inherit settings from SERIAL but don't use it in visualization.
        inherit = None, SERIAL
\end{lstlisting}

\subsection{Generating Tasks Automatically}
\label{Generating Tasks}

Groups of tasks that are closely related such as an ensemble of model runs or 
a family of obs processing tasks, or sections of workflow that are repeated
with minor variations, can be generated automatically by iterating over
some integer range (e.g.\ \lstinline=model<n>= for \lstinline@n = 1..10@) or
list of strings (e.g.\ \lstinline=obs<type>= for
\lstinline@type = ship, buoy, radiosonde, ...@).

\subsubsection{Jinja2 Loops}

Task generation was traditionally done in Cylc with explicit Jinja2 loops,
like this:
\lstset{language=suiterc}
\begin{lstlisting}
# Task generation the old way: Jinja2 loops (NO LONGER RECOMMENDED!)
{% set PARAMS = range(1,11) %}
[scheduling]
    [[dependencies]]
        graph = """
{% for P in PARAMS %}
      pre => model_p{{P}} => post
      {% if P == 5 %}
          model_p{{P}} => check
      {% endif %}
{% endfor %}    """
[runtime]
{% for P in PARAMS %}
    [[model_p{{P}}]]
        script = echo "my parameter value is {{P}}"
    {% if P == 1 %}
        # special case...
    {% endif %}
{% endfor %}
\end{lstlisting}

Unfortunately this makes a mess of the suite definition, particularly the
scheduling graph, and it gets worse with nested loops over multiple parameters.

\includegraphics[width=\textwidth]{resources/png/param-1.png}

\subsubsection{Parameterized Tasks}
\label{Parameterized Tasks}

Cylc-6.11 introduced built-in {\em suite parameters} for generating tasks
without destroying the clarity of the base suite definition. Here's the same
example using suite parameters instead of Jinja2 loops:

\lstset{language=suiterc}
\begin{lstlisting}
# Task generation the new way: suite parameters.
[cylc]
    [[parameters]]
        p = 1..10
[scheduling]
    [[dependencies]]
        graph = """pre => model<p> => post
                model<p=5> => check"""
[runtime]
    [[model<p>]]
        script = echo "my parameter value is ${CYLC_TASK_PARAM_p}"
    [[model<p=7>]]
        # special case ...
\end{lstlisting}

Here \lstinline@model<p>@ expands to \lstinline@model_p7@ for \lstinline@p=7@,
and so on, via the default expansion template for integer-valued parameters,
but custom templates can be defined if necessary. Parameters can also be
defined as lists of strings, and you can define dependencies between different
values: \lstinline@chunk<p-1> => chunk<p>@.  Here's a multi-parameter example:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = a, b, c
        m = 1..5
[scheduling]
    [[dependencies]]
        graph = pre => init<run> => sim<run,m> => close<run> => post
[runtime]
    [[sim<run,m>]]
\end{lstlisting}

\includegraphics[width=\textwidth]{resources/png/param-2.png}

If family members are defined by suite parameters, then parameterized
trigger expressions are equivalent to family \lstinline=:<state>-all= triggers.
For example, this:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[parameters]]
        n = 1..5
[scheduling]
    [[dependencies]]
        graph = pre => model<n> => post
[runtime]
    [[MODELS]]
    [[model<n>]]
        inherit = MODELS
\end{lstlisting}

is equivalent to this:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[parameters]]
        n = 1..5
[scheduling]
    [[dependencies]]
        graph = pre => MODELS:succeed-all => post
[runtime]
    [[MODELS]]
    [[model<n>]]
        inherit = MODELS
\end{lstlisting}

(but future plans for family triggering may make the second case more
efficient for very large families).

For more information on parameterized tasks see the Cylc user guide.

\subsection{Optional App Config Files}
\label{Optional App Config Files}

Closely related tasks with few configuration differences between them - such as
multiple UM forecast and reconfiguration apps in the same suite - should use
the same Rose app configuration with the differences supplied by optional
configs, rather than duplicating the entire app for each task.

Optional app configs should be valid on top of the main app config and not
dependent on the use of other optional app configs. This ensures they will
work correctly with macros and can therefore be upgraded automatically.

\note{Currently optional configs don't work very well with UM STASH
  configuration - see Section~\ref{UM STASH in Optional App Configs}.}

Optional app configs can be loaded by command line switch:

\begin{lstlisting}
rose task-run -O key1 -O key2
\end{lstlisting}

or by environment variable:

\begin{lstlisting}
ROSE_APP_OPT_CONF_KEYS = key1 key2
\end{lstlisting}

The environment variable is generally preferred in suites because you don't
have to repeat and override the root-level script configuration: 

\begin{lstlisting}
[runtime]
    [[root]]
        script = rose task-run -v
    [[foo]]
        [[[environment]]]
            ROSE_APP_OPT_CONF_KEYS = key1 key2
\end{lstlisting}
