JSON Schema for a Comet Data Pipeline. Json Format

Edit JSON Schema for a Comet Data Pipeline.

types types

#	name	primitiveType	pattern	zone	sample	comment	indexMapping	Actions

.env

_

.transform

_

transform.name - Job name. Must be set to the prefix of the filename. [JOBNAME].comet.yml

transform.area - Area where the data is located.\nWhen using the BigQuery engine, the area corresponds to the dataset name we will be working on in this job.\nWhen using the Spark engine, this is folder where the data should be store. Default value is "business"

transform.format - output file format when using Spark engine. Ignored for BigQuery. Default value is "parquet"

transform.coalesce - When outputting files, should we coalesce it to a single file. Useful when CSV is the output format.

transform.udf - Register UDFs written in this JVM class when using Spark engine.\nRegister UDFs stored at this location when using BigQuery engine

transform.tasks tasks

#	sql	engine	domain	dataset	write	area	sink.type	sink.name	sink.id	sink.timestamp	Actions

.transform.views

_

transform.engine - SPARK or BQ. Default value is SPARK.

.load

_

load.name - Domain name. Make sure you use a name that may be used as a folder name on the target storage. - When using HDFS or Cloud Storage, files once ingested are stored in a sub-directory named after the domain name. - When used with BigQuery, files are ingested and sorted in tables under a dataset named after the domain name.

load.directory - Folder on the local filesystem where incoming files are stored. Typically, this folder will be scanned periodically to move the dataset to the cluster for ingestion. Files located in this folder are moved to the pending folder for ingestion by the "import" command.

.load.metadata

_

.load.metadata.partition

_

load.metadata.partition.sampling - 0.0 means no sampling, > 0 && < 1 means sample dataset, >=1 absolute number of partitions.

load.metadata.mode - FILE mode by default.\nFILE and STREAM are the two accepted values.\nFILE is currently the only supported mode.

load.metadata.format - DSV by default. Supported file formats are :\n- DSV : Delimiter-separated values file. Delimiter value is specified in the "separator" field.\n- POSITION : FIXED format file where values are located at an exact position in each line.\n- SIMPLE_JSON : For optimisation purpose, we differentiate JSON with top level values from JSON\n with deep level fields. SIMPLE_JSON are JSON files with top level fields only.\n- JSON : Deep JSON file. Use only when your json documents contain subdocuments, otherwise prefer to\n use SIMPLE_JSON since it is much faster.\n- XML : XML files

load.metadata.encoding - UTF-8 if not specified.

load.metadata.multiline - are json objects on a single line or multiple line ? Single by default. false means single. false also means faster

load.metadata.array - Is the json stored as a single object array ? false by default. This means that by default we have on json document per line.

load.metadata.withHeader - does the dataset has a header ? true bu default

load.metadata.separator - the values delimiter, ';' by default value may be a multichar string starting from Spark3

load.metadata.quote - The String quote char, '"' by default

load.metadata.escape - escaping char '\' by default

load.metadata.write - Append to or overwrite existing data

load.metadata.partition.attributes attributes

#	Values	Actions

.load.metadata.sink

_

load.metadata.sink.type - Where to sink the data

load.metadata.sink.name - Once ingested, files may be sinked to BigQuery, Elasticsearch or any JDBC compliant Database.

load.metadata.sink.id - ES: Attribute to use as id of the document. Generated by Elasticsearch if not specified.

load.metadata.sink.timestamp - BQ: The timestamp column to use for table partitioning if any. No partitioning by default\nES:Timestamp field format as expected by Elasticsearch ("{beginTs|yyyy.MM.dd}" for example).

load.metadata.sink.location - BQ: Database location (EU, US, ...)

load.metadata.sink.clustering - BQ: List of ordered columns to use for table clustering

load.metadata.sink.days - BQ: Number of days before this table is set as expired and deleted. Never by default.

load.metadata.sink.requirePartitionFilter - BQ: Should be require a partition filter on every request ? No by default.

load.metadata.sink.connection - JDBC: Connection String

load.metadata.sink.partitions - JDBC: Number of Spark partitions

load.metadata.sink.batchSize - JDBC: Batch size of each JDBC bulk insert

load.metadata.ignore - Pattern to ignore or UDF to apply to ignore some lines

load.metadata.clustering List of attributes to use for clustering

#	Values	Actions

.load.metadata.xml

_

load.comment - Domain Description (free text)

load.ack - Ack extension used for each file. ".ack" if not specified. Files are moved to the pending folder only once a file with the same name as the source file and with this extension is present. To move a file without requiring an ack file to be present, set explicitly this property to the empty string value "".

load.schemaRefs List of files containing the schemas. Should start with an '_' and be located in the same folder.

#	Values	Actions

load.schemas List of schemas for each dataset in this domain. A domain usually contains multiple schemas. Each schema defining how the contents of the input file should be parsed. See Schema for more details.

#	name	pattern	metadata.partition.sampling	metadata.mode	metadata.format	metadata.encoding	metadata.multiline	metadata.array	metadata.withHeader	metadata.separator	Actions

load.extensions recognized filename extensions. json, csv, dsv, psv are recognized by default. Only files with these extensions will be moved to the pending folder.

#	Values	Actions

.assertions

_

.views

_

schemas List of schemas for each dataset in this domain. A domain usually contains multiple schemas. Each schema defining how the contents of the input file should be parsed. See Schema for more details.

#	name	pattern	metadata.partition.sampling	metadata.mode	metadata.format	metadata.encoding	metadata.multiline	metadata.array	metadata.withHeader	metadata.separator	Actions

ToolMatter

Your Apps in Action

Use this form to visualize JSON Schema for a Comet Data Pipeline..