Ingestion
DataHub ingestion
type: "io.kestra.plugin.datahub.Ingestion"Run DataHub ingestion
id: datahub_cli
namespace: company.name
tasks:
- id: cli
type: io.kestra.plugin.datahub.Ingestion
recipe:
source:
type: mysql
config:
host_port: 127.0.0.1:3306
database: dbname
username: root
password: "{{ secret('MYSQL_PASSWORD') }}"
sink:
type: datahub-rest
config:
server: http://datahub-gms:8080
Run DataHub ingestion using local recipe file
id: datahub_cli
namespace: company.name
tasks:
- id: cli
type: io.kestra.plugin.datahub.Ingestion
recipe: "{{ input('recipe_file') }}"
NOThe Ingestion DataHub Recipe.
YESacryldata/datahub-ingestion:headThe Ingestion DataHub docker image.
YESThe environments for Ingestion DataHub.
YESThe files to create on the local filesystem. It can be a map or a JSON object.
NOInject namespace files.
Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.
YESThe files from the local filesystem to send to Kestra's internal storage.
Must be a list of glob expressions relative to the current working directory, some examples: my-dir/**, my-dir/*/** or my-dir/my-file.txt.
NO{
"type": "io.kestra.plugin.scripts.runner.docker.Docker"
}The task runner to use.
0The exit code of the entire flow execution.
The output files' URIs in Kestra's internal storage.
The value extracted from the output of the executed commands.
YEStrueWhether to enable namespace files to be loaded into the working directory. If explicitly set to true in a task, it will load all Namespace Files into the task's working directory. Note that this property is by default set to true so that you can specify only the include and exclude properties to filter the files to load without having to explicitly set enabled to true.
YESA list of filters to exclude matching glob patterns. This allows you to exclude a subset of the Namespace Files from being downloaded at runtime. You can combine this property together with include to only inject a subset of files that you need into the task's working directory.
YESOVERWRITEOVERWRITEFAILWARNIGNOREComportment of the task if a file already exist in the working directory.
YESA list of filters to include only matching glob patterns. This allows you to only load a subset of the Namespace Files into the working directory.
YES["{{flow.namespace}}"]A list of namespaces in which searching files. The files are loaded in the namespace order, and only the latest version of a file is kept. Meaning if a file is present in the first and second namespace, only the file present on the second namespace will be loaded.
NO\d+\.\d+\.\d+(-[a-zA-Z0-9-]+)?|([a-zA-Z0-9]+)The version of the plugin to use.