Avionix Airflow Options

avionix_airflow’s configuration can be changed by using option objects. For the local implementation most settings will not need to be changed, but for cloud implementations many will likely need to be set, in particular the AwsOptions and AirflowOptions.

class avionix_airflow.AirflowOptions(dag_sync_image, dag_sync_command, dag_sync_schedule, dag_storage='50Mi', log_storage='50Mi', external_storage='50Mi', default_executor_cpu=5, default_executor_memory=2, access_modes=None, default_timezone='utc', core_executor='CeleryExecutor', namespace='airflow', domain_name='www.avionix-airflow.com', additional_vars=None, fernet_key='', dags_paused_at_creation=True, worker_image='', worker_image_tag='', open_node_ports=False, local_mode=False, smtp_notification_options=None, git_ssh_key=None, image_pull_policy='IfNotPresent', master_image='', master_image_tag='', delete_pods_on_failure=False)

Class for storing airflow options. Note that for storage specification the default unit is bytes, and you can use Mi or Gi to specify larger units.

Parameters

  • dag_sync_image (str) – The image to use for syncing dags

  • dag_sync_command (List[str]) – The commands to use for syncing dags

  • dag_sync_schedule (str) – Cron format schedule for how often to sync dags

  • dag_storage (str) – How much persistent storage to allocate for dags

  • logs_storage – How much persistent storage to allocate for dags

  • external_storage (str) – How much external storage to allocate for dags

  • default_executor_cpu (int) – Default CPU to allocate to the executor This is specified in number of vCPUs. You can fractions of CPUs as well or milliCPUs (eg. “500m” for 500 milliCPUs)

  • default_executor_memory (int) – Default memory to allocate to the executor This can be specified in bytes (default) or using standard prefix (eg. “20Mi”)

  • access_modes (Optional[InitVar]) – Used to set access modes on the dag and logs volumes

  • default_timezone (str) – The default timezone

  • core_executor (str) – The core executor to use, supported executors are “KubernetesExecutor” and “CeleryExecutor”

  • namespace (str) – The kubernetes namespace to use for the installation

  • domain_name (Optional[str]) – The domain name to place on the ingress controller

  • additional_vars (Optional[InitVar]) – Any additional environment variables to place on the airflow nodes. Note that you can add additional airflow configuration using this setting. For more info on this see the airflow config documentation

  • fernet_key (InitVar) – The fernet key to use for airflow, this is autogenerated by default, so unless you need to use an old fernet key, just leave this

  • dags_paused_at_creation (bool) – Whether or not dags should be paused on creation

  • worker_image (str) – The docker image to use as the airflow worker

  • worker_image_tag (str) – The docker tag of the airflow worker image

  • open_node_ports (bool) – Whether to expose the node ports

  • local_mode (bool) – Whether or not to run in local mode

  • image_pull_policy (str) – Policy for pulling docker image. Can be one of “Never”, “Always”, “IfNotPresent”

  • master_image_tag (str) – The docker tag to use for the master image

  • delete_pods_on_failure (bool) – Whether or not to terminate pods if KubernetesExecutor task fails

class avionix_airflow.AwsOptions(efs_id, cluster_name, elastic_search_access_role_arn, default_role_arn, alb_role_arn, external_dns_role_arn, autoscaling_role_arn, dag_sync_role_arn, domain, domain_filters=None, use_ssl=False)

Settings for the AWS Managed Node Group setup

Parameters

  • efs_id (str) – The id of the EFS file system to use as the storage provider

  • cluster_name (str) – The name of the EKS cluster

  • elastic_search_access_role_arn (str) – The IAM role arn for accessing elastic search

  • default_role_arn (str) – The default IAM role arn for pods in the cluster

  • alb_role_arn (str) – The IAM role setting up application load balancing

  • external_dns_role_arn (str) – The IAM role for setting the domain name

  • autoscaling_role_arn (str) – The IAM role for controlling the cluster scaling

  • domain (str) – The AWS domain name to use

  • dag_sync_role_arn (str) – The IAM role use for controlling retrieval of dags

  • domain_filters (Optional[List[str]]) – A list used to filter out route53 domain names

  • use_ssl (bool) – Whether or not to use SSL encryption, (Recommended to be True)

class avionix_airflow.MonitoringOptions(enabled=True, elastic_search_uri='http://elasticsearch-master:9200', grafana_role='Viewer', elastic_search_proxy_uri='elasticsearch-master')

Configurations for monitoring airflow

Parameters

  • enabled (bool) – Whether or not monitoring should be enabled

  • elastic_search_uri (str) – The uri to use for elastic search

  • grafana_role (str) – The role to use for grafana can be one of [“Viewer”, “Editor”, “Admin”]

  • elastic_search_proxy_uri (str) – The uri to use as the elastic search proxy, this shouldn’t be changed unless using an external elastic search provider

class avionix_airflow.RedisOptions(port=6379, host='redis-svc', proto='redis://', password='', db_num=1)

Configuration for redis if using “CeleryExecutor” option in AirflowOptions

Parameters

  • port (int) – The port to use for redis

  • host (str) – Name of the redis host, defaults to the internal service name

  • proto (str) – Redis protocol

  • password (str) – Redis password

  • db_num (int) – Redis db number

class avionix_airflow.SqlOptions(user='airflow', password='airflow', host='airflow-database-connection', port=5432, db='airflow', create_database_in_cluster=True, extras='')

Provides configuration for the airflow backend database. Currently only postgres is supported. Note that all connection related strings will be stored in kubernetes secrets on the cluster.

Parameters

  • user (str) – The username for database

  • password (str) – The pass for the database

  • host (str) – The hostname for the database

  • port (int) – The port to keep open for the database

  • db (str) – The database name

  • create_database_in_cluster (bool) – Whether airflow should create the database

  • extras – Any extra params to be appended to the end of the db connection string