datamineris a MTConnect Agent DataConnect Gateway process. It streams data from any MTConnect Agent 1.4+ to data services for RethinkDB, InfluxDB, ElasticSearch, or Kafak.
agents:
#
# MTConnect agent info
#
- name: Mazak-5719 # name is used to create data ids in the data services
url: http://mtconnect.mazakcorp.com:5719 # MTConnect ReST url connection info
freq: 10 # data polling frequence in seconds
# Repeat the section for additional MTConnect agent
#
- name: demo
url: https://demo.metalogi.io
freq: 1
# DataConnect configuration
server:
# =================== RethinkDB Defaults ==========================
#
# this section should be removed if RethinkDB is not required
#
RethinkDBServer: localhost:28015 # default is none (RethinkDB interface is disabled)
#RethinkDBAuthKey: <your authKey if needed>
RethinkDBDatabase: testing # default is mtconnect. RethinkDB Database where all the tables are located
RethinkDBProbeTable: mtprobe # default is none (disabled)
RethinkDBItemInfoTable: itemInfo # default is none (disabled)
RethinkDBNodeTreeTable: sourceinfo # default is none (disabled)
RethinkDBDataTable: mtdata # default is mtdata
RethinkDBQueueSize: 20000 # default is 10000, max # of queue items.
RethinkDBBatchSize: 100 # default is 200, number of records to be processed together
# =================== InfluxDB Defaults ==========================
#
# this section should be removed if InfluxDB is not required
#
InfluxDBUrl: https://localhost:8086 # default is none (InfluxDB interface is disabled)
InfluxDBOrganization: testing # InfluxDB Organization where the bucket is located
InfluxDBBucket: test # InfluxDB Bucket
#InfluxDBAuthKey: <your authKey if needed>
InfluxDBQueueSize: 20000 # default is 10000, max # of queue items.
InfluxDBBatchSize: 100 # default is 200, number of records to be processed together
# =================== ElasticSearch Defaults ==========================
#
# this section should be removed if ElasticSearch is not required
#
ElasticSearchUrl: http://localhost:9200 # default is none (ElasticSearch interface is disabled)
ElasticSearchUser: YourUserId
ElasticSearchPassword: YourPassword
ElasticSearchIndexName: mtconnect # default is mtconnect, Index name associated with the data.
ElasticSearchQueueSize: 20000 # default is 10000, max # of queue items.
ElasticSearchBatchSize: 100 # default is 200, number of records to be processed together
# =================== Kafka Defaults ==========================
#
# this section should be removed if Kafka is not required
#
KafkaBrokers: localhost:9093 # default is none (Kafka interface is disabled)
KafkaTopicDeviceProbe: MTConnect.NewDeviceProbe # default is none (disabled)
KafkaTopicDeviceItemInfo: MTConnect.NewDeviceItemInfo # default is none (disabled)
KafkaTopicDeviceNodeTree: MTConnect.NewDeviceNodeTree # default is none (disabled)
KafkaTopicDeviceData: MTConnect.NewDeviceData # default is MTConnect.DeviceData
KafkaQueueSize: 20000 # default is 10000, max # of queue items.
KafkaBatchSize: 100 # default is 200, number of records to be processed together
#
# track last sequence number for each MTAgent, use it to recover on startup
# default is $HOME/dataminer.settings
recoveryStore: ./dataminer.settings
# recovery mode can be first, current or last
# first and current are the sequence number retrieved from MT Agents
# last is determined from the recoveryStore
#
# default is last
recoveryMode: current
# license
#licenseKey: <your Metalogi license key>
# serverity can be debug, info, warn, error or off
logging:
severity: info
agents
name : The device name. This is used to form the data IDs in the data services.url / frequency : Specify the MTConnect agent's URL and how often to poll the data.server
recoveryStore : track the last sequence number for each MTAgent, use it to recover on startup, the default is $HOME/dataminer.settings.recoveryMode : recovery mode can be first, current or last. first and currentare to use the sequence number retrieved from the MTConnect Agent. last is determined by the recoveryStore.RethinkDBServer : RethinkDB client driver connection info in <host name>:<port #>RethinkDBDatabase : Target Rethink DB database nameRethinkDBAuthKey : RethinkDB connection authorization keyRethinkDBProbeTable : Specify the table to store the metadata of the Agent's machine, the default is None (Disabled).RethinkDBNodeTreeTable : Specify the table to store the unified data selection tree from the metadata, the default is None (Disabled).RethinkDBItemInfoTable : Specify the table to store the type info for each data item, the default is None (Disabled).RethinkDBAssetInfoTable : Specify the table to store the asset info in the Agent's data stream, the default is None (Disabled)RethinkDBDataTable : Specify the table to store the Samples, Events, and Condition real-time data, the default is mtconnect.RethinkDBQueueSize : max # of queue items, the default is 10000.RethinkDBBatchSize : Specify # of records to be processed in a batch, the default is 200.InfluxDBUrl : The InfluxDB HTTP service's URLInfluxDBOrganization : Organization name - default is mtconnect.InfluxDBBucket : Bucket name - default is mtconnect.InfluxDBAuthKey : InfluxDB's authorization key info.InfluxDBQueueSize : max # of queue items, the default is 10000.InfluxDBBatchSize : Specify # of records to be processed in a batch, the default is 200.ElasticSearchUrl : Elasticsearch API service's URL.ElasticSearchUser/ElasticSearchPassword : Elasticsearch service login credential.ElasticSearchIndexName : The index name for the bulk insert data, default is mtconnect.ElasticSearchBatchSize : Specify # of records for each bulk insert, default is 200.KafkaBrokers : One or more Kafka Brokers can be specified as a list of <host>:<port #> and separated by a comma.KafkaTopicDeviceProbe : The topic name on which the metadata of the Agent's machine is published, default is None (disabled).KafkaTopicDeviceNodeTree : The topic name for publishing the unified data selection tree constructed from the metadata, default is None (disabled).KafkaTopicDeviceItemInfo : The topic name for publishing the type info for each data item, default is None (disabled).KafkaTopicDeviceAsset : The topic name for publishing the asset info in the Agent's data stream, default is None (disabled).KafkaTopicDeviceData : The topic name for publishing the Samples, Events, and Condition real-time data, default is MTConnect.DeviceData.KafkaQueueSize : max # of queue items, the default is 10000.KafkaBatchSize : Specify how many data items are to be batched for each publication, default is 200.
C:\Program Files\MetaAgent\examples\dataminer>dataminer dataminer.cfg
Metalogi MetaAgent - Dataminer 2.2.0.600 - built on Wed Oct 11 22:28:27 2023
[2023-10-30T07:37:31] info Settings are loaded from dataminer.cfg
[2023-10-30T07:37:31] info ----------------
[2023-10-30T07:37:31] info Name: Mazak-5719
[2023-10-30T07:37:31] info Agent Uri: http://mtconnect.mazakcorp.com:5719
[2023-10-30T07:37:31] info Poll Interval: 10
[2023-10-30T07:37:31] info Next Sequence #:
[2023-10-30T07:37:31] info RethinkDB Connect Info: localhost:28015 database:testing
[2023-10-30T07:37:35] info Product RethinkDB Sink is verified
[2023-10-30T07:37:35] info Product InfluxDB Sink is verified
[2023-10-30T07:37:35] info InfluxDB connection info: http://localhost:8086 Organization: testing Bucket: test
[2023-10-30T07:37:39] info ElasticSearch connection info: http://localhost:9200 index: mtconnect
[2023-10-30T07:37:43] info Kakfa connection info: brokers localhost:9093
[2023-10-30T07:37:43] info Product Kafka Sink is verified
[2023-10-30T07:37:43] info ----------------
[2023-10-30T07:37:43] info Name: demo
[2023-10-30T07:37:43] info Agent Uri: https://demo.metalogi.io
[2023-10-30T07:37:43] info Poll Interval: 1
[2023-10-30T07:37:43] info Next Sequence #:
[2023-10-30T07:37:46] info # of KafkaMgr records sent = 0