Managing InfluxDB Sink connectors¶
An InfluxDB Sink connector consumes data from Kafka and writes to InfluxDB. Sasquatch uses the Telegraf Kafka consumer input plugin and the InfluxDB v1 output plugin implemented in the telegraf-kafka-consumer subchart.
Configuration¶
The connector configuration is specified per Sasquatch environment in sasquatch/values-<environment>.yaml
.
Here’s what the connector configuration for writing data from the lsst.example.skyFluxMetric
kafka topic to InfluxDB looks like:
telegraf-kafka-consumer:
enabled: true
kafkaConsumers:
example:
enabled: true
topicRegexps: |
[ "lsst.example" ]
database: "lsst.example"
timestamp_field: "timestamp"
timestamp_format: "unix_ms"
tags: |
[ "band", "instrument" ]
replicaCount: 1
The following sections cover the most important configuration options using the lsst.example.skyFluxMetric
metric as an example.
See the telegraf-kafka-consumer subchart for the configuration options and default values.
See the Avro schemas section to learn more about the lsst.example.skyFluxMetric
example in Sasquatch.
Selecting Kafka topics¶
kafkaConsumers.example.topicRegexps
is a list of regular expressions used to specify the Kafka topics consumed by this connector, and KafkaConsumers.example.database
is the name of the InfluxDB v1 database to write to.
In this example, all Kafka topics prefixed by lsst.example
are recorded in the lsst.example
database in InfluxDB.
Note
If the database doesn’t exist in InfluxDB it is automatically create by Telegraf. Telegraf also records internal metrics from its input and output plugins in the same database.
Timestamp¶
InfluxDB, being a time-series database, requires a timestamp to index the data.
The name of the field that contains the timestamp value and the timestamp format are specified by the kafkaConsumers.example.timestamp_field
and
kafkaConsumers.timestamp_format
keys.
Operations¶
Deployment¶
To deploy a connector sync the connector ConfigMap and Deployment Kubernetes resources in Argo CD.
List, stop and start connectors¶
To list the connectors in a given Sasquatch environment, run:
kubectl get deploy -l app.kubernetes.io/name=sasquatch-telegraf -n sasquatch
To view the view the logs of a single connector instance, run:
kubectl logs -l app.kubernetes.io/instance=sasquatch-telegraf-<connector-name> -n sasquatch
To stop the connectors you can scale the deployment replicas down to zero:
kubectl scale deploy -l app.kubernetes.io/name=sasquatch-telegraf --replicas=0 -n sasquatch
To start the connectors you can scale the deployment replicas back to one:
kubectl scale deploy -l app.kubernetes.io/name=sasquatch-telegraf --replicas=1 -n sasquatch
To permanently remove a connector set the kafkaConsumers.<connector name>.enabled
key to false
in the sasquatch/values-<environment>.yaml
file and sync the connector ConfigMap and the Deployment Kubernetes resources in Argo CD.
Monitoring¶
Telegraf internal metrics are recorded under the telegraf
database in Sasquatch and provide information about memory and buffer usage, throughput as well as read and write errors for each connector instance.