nc-link-dataminer is an NC-Link DataConnect Gateway process. It streams data from any NC-Link services to data services for RethinkDB, InfluxDB, ElasticSearch, or Kafak.
# # devices
devices:
- name: d1 # name is used to create data ids in the data services
uuid: d1 # device identifier in the NC-Link service
staleInterval: 1000 # wait staleInterval (in milliseconds) before sending updates. <0 - update only when changed.
#
# Repeat the section for additional device
#
- name: d2
uuid: d2
#
# NC-Link MQTT Agent information
#
server:
name: Demo
mqtt: demo.metalogi.io
port: 1883
# =================== RethinkDB Defaults ==========================
#
# this section should be removed if RethinkDB is not required
#
RethinkDBServer: localhost:28015
#RethinkDBAuthKey: <your authKey if needed>
RethinkDBDatabase: testing # default is mtconnect
RethinkDBNodeTreeTable: sourceinfo # default is none (disabled)
RethinkDBDataTable: mtdata # default is mtdata
RethinkDBQueueSize: 20000 # default is 10000, max # of queue items.
RethinkDBBatchSize: 100 # default is 200, number of records to be processed together
# =================== InfluxDB Defaults ==========================
#
# this section should be removed if InfluxDB is not required
#
InfluxDBUrl: http://localhost:8086
InfluxDBOrganization: YourOrganization # InfluxDB Organization where the bucket is located
InfluxDBBucket: YourBucket # InfluxDB Bucket
#InfluxDBAuthKey: <your authKey if needed>
InfluxDBOrganization: testing
InfluxDBQueueSize: 20000 # default is 10000, max # of queue items.
InfluxDBBatchSize: 100 # default is 200, number of records to be processed together
# =================== ElasticSearch Defaults ==========================
#
# this section should be removed if ElasticSearch is not required
#
ElasticSearchUrl: http://localhost:9200
ElasticSearchUser: YourUserId
ElasticSearchPassword: YourPassword
ElasticSearchIndexName: mtconnect # default is mtconnect, Index name associated with the data.
ElasticSearchQueueSize: 20000 # default is 10000, max # of queue items.
ElasticSearchBatchSize: 100 # default is 200, number of records to be processed together
# =================== Kafka Defaults ==========================
#
# this section should be removed if Kafka is not required
#
KafkaBrokers: localhost:9092
KafkaTopicDeviceNodeTree: MTConnect.NewDeviceNodeTree # default is none (disabled)
KafkaTopicDeviceData: MTConnect.NewDeviceData # default is MTConnect.DeviceData
KafkaQueueSize: 20000 # default is 10000, max # of queue items.
KafkaBatchSize: 100 # default is 200, number of records to be processed together
# license
#licenseKey: <your Metalogi license key>
# serverity can be debug, info, warn, error or off
logging:
severity: info
devices : a list of name and uuid pairs. name is used to create data IDs in the data services. uuid is the NC-Link device ID and it is used to identify and capture the device's data in the MQTT stream data.server
mqtt : MQTT server's hostname or IP addressport : MQTT server's port #user / password : MQTT login credential if requiredcacert : The location of CA certificate file.useWebSocket : Use web socket, default is false.RethinkDBServer : RethinkDB client driver connection info in <host name>:<port #>RethinkDBDatabase : Target Rethink DB database nameRethinkDBAuthKey : RethinkDB connection authorization keyRethinkDBProbeTable : Specify the table to store the metadata of the Agent's machine, the default is None (Disabled).RethinkDBNodeTreeTable : Specify the table to store the unified data selection tree from the metadata, the default is None (Disabled).RethinkDBItemInfoTable : Specify the table to store the type info for each data item, the default is None (Disabled).RethinkDBAssetInfoTable : Specify the table to store the asset info in the Agent's data stream, the default is None (Disabled)RethinkDBDataTable : Specify the table to store the Samples, Events, and Condition real-time data, the default is mtconnect.RethinkDBQueueSize : max # of queue items, the default is 10000.RethinkDBBatchSize : Specify # of records to be processed in a batch, the default is 200.InfluxDBUrl : The InfluxDB HTTP service's URLInfluxDBOrganization : Organization name - default is mtconnect.InfluxDBBucket : Bucket name - default is mtconnect.InfluxDBAuthKey : InfluxDB's authorization key info.InfluxDBQueueSize : max # of queue items, the default is 10000.InfluxDBBatchSize : Specify # of records to be processed in a batch, the default is 200.ElasticSearchUrl : Elasticsearch API service's URL.ElasticSearchUser/ElasticSearchPassword : Elasticsearch service login credential.ElasticSearchIndexName : The index name for the bulk insert data, default is mtconnect.ElasticSearchBatchSize : Specify # of records for each bulk insert, default is 200.KafkaBrokers : One or more Kafka Brokers can be specified as a list of <host>:<port #> and separated by a comma.KafkaTopicDeviceProbe : The topic name on which the metadata of the Agent's machine is published, default is None (disabled).KafkaTopicDeviceNodeTree : The topic name for publishing the unified data selection tree constructed from the metadata, default is None (disabled).KafkaTopicDeviceItemInfo : The topic name for publishing the type info for each data item, default is None (disabled).KafkaTopicDeviceAsset : The topic name for publishing the asset info in the Agent's data stream, default is None (disabled).KafkaTopicDeviceData : The topic name for publishing the Samples, Events, and Condition real-time data, default is MTConnect.DeviceData.KafkaQueueSize : max # of queue items, the default is 10000.KafkaBatchSize : Specify how many data items are to be batched for each publication, default is 200.
C:\Program Files\MetaAgent\examples\nc-link-dataminer>nc-link-dataminer nc-link-dataminer.cfg
Metalogi MetaAgent - NC-Link Dataminer 2.2.0.600 - built on Wed Oct 11 22:28:27 2023
[2023-10-30T08:23:16] info Settings are loaded from nc-link-dataminer.cfg
[2023-10-30T08:23:16] info Product RethinkDB Sink is verified
[2023-10-30T08:23:16] info Product InfluxDB Sink is verified
[2023-10-30T08:23:16] info Product ElasticSearch Sink is verified
[2023-10-30T08:23:16] info Product Kafka Sink is verified
[2023-10-30T08:23:16] info RethinkDB Connect Info: localhost:28015 database:testing
[2023-10-30T08:23:20] info InfluxDB connection info: http://localhost:8086 Organization: YourOrganization Bucket: YourBucket
[2023-10-30T08:23:24] info ElasticSearch connection info: http://localhost:9200 index: mtconnect
[2023-10-30T08:23:28] info Kakfa connection info: brokers localhost:9092
[2023-10-30T08:23:28] info Connecting adapter to MQTT/TCP Broker on demo.metalogi.io:1883
[2023-10-30T08:23:28] info # of KafkaMgr records sent = 0
[2023-10-30T08:23:28] info device [d1] d1 - stale data update interval 1000
[2023-10-30T08:23:28] info device [d2] d2 - stale data update interval -1
[2023-10-30T08:23:29] info publish received "Probe/Query/Response/d1" size 34699 - process device's model file.
[2023-10-30T08:23:29] info publish received "Probe/Query/Response/d2" size 42 - process device's model file.
[2023-10-30T08:23:31] info async_connect callback: Success
[2023-10-30T08:23:31] info publish received "Query/Response/d1" size 2078 - process device's query result.
[2023-10-30T08:23:33] info publish received "Sample/d1/EVENT_CHANNEL" size 3348
[2023-10-30T08:23:33] info publish received "Sample/d1/CONDITION_CHANNEL" size 7655
[2023-10-30T08:23:33] info publish received "Sample/d1/SAMPLE_CHANNEL" size 1489
[2023-10-30T08:23:37] info publish received "Sample/d1/SAMPLE_CHANNEL" size 1506