Introduction

ctrl is a Command & Control (C2) backend system for Servers, IOT and Edge platforms. Simply put, control anything like a Fleet Manager.

In operations ctrl can be used to send a set of commands to execute to a node, a group of nodes, or all nodes. It can also be used to gather operational data like logs and metrics from all the ctrl nodes.

In forensics ctrl can be used to execute commands with tools to gather information needed. If the tools are not already present on an end ctrl node, ctrl can be used to copy the tools to that node, and then ctrl can run them to gather information.

Usecases

Some example usecases are:

• Send shell commands or scripts to control one or many end nodes that will instruct to change config, restart services and control those systems.
• Gather data from both secure and not secure devices and systems, and transfer them encrypted in a secure way over the internet to your central system for handling those data.
• Collect metrics or monitor end nodes, then send and store the result to some ctrl instance, or pass those data's on to another ctrl instance for further handling of metrics or monitoring data.
• Distribute certificates.
• Run as a sidecar in Kubernetes for direct access to the pod.

Ctrl is a versatile tool that enables users to control their systems with precision and efficiency. It's designed as a fleet manager, leveraging NATS as its messaging architecture, allowing for secure and encrypted communication between an operator and one or more servers.

With Ctrl, you can accomplish tasks that you typically perform on a shell in a system - all with simple messages, using Ctrl's request methods. This feature allows for the execution of commands on remote servers, even if they are down at the time of sending. Messages will be retried based on the specified criteria in their body.

Ctrl is built to handle multiple messages independently, by utilizing Go, the programming languages builtin concurrency. It can handle tasks such as executing a slow process without affecting other processes or systems. If a process fails, Ctrl ensures that it doesn't affect other operations, providing a reliable and robust system control solution.

Furthermore, Ctrl is highly compatible with various host operating systems. It supports cloud containers, Raspberry Pi, and any other device with an installed operating system. It can run on a variety of architectures such as x86, amd64, arm64, ppc64, making it a versatile tool for system administration.

Ctrl is compatible with most major operating systems including Linux, OSX, and Windows, giving users the flexibility to use Ctrl across different platforms.

Install with docker

Start up a local nats message broker

docker run -p 4444:4444 nats -p 4444

Create a ctrl docker image.

git clone git@github.com:postmannen/ctrl.git
cd ctrl
docker build -t ctrl:test1 .

Create a folder which will be the working directory for the node. This is where we keep the .env file, and can mount local host folders to folders within the container.

mkdir -p testrun/readfolder
cd testrun

create a .env file

cat << EOF > .env
NODE_NAME="node1"
BROKER_ADDRESS="127.0.0,1:4444"
NKEY_SEED=<REPLACE WITH seed created for the node>
ENABLE_DEBUG=1
START_PUB_HELLO=60
IS_CENTRAL_ERROR_LOGGER=0
EOF

Start the ctrl container. To be able to send messages into ctrl we mount the readfolder to a local directory. When we later got a messages to send we just copy it into the read folder and ctrl will pick it up and handle it. Messages can be in either YAML or JSON format.

docker run --env-file=".env" --rm -ti -v $(PWD)/readfolder:/app/readfolder ctrl:test1

Prepare and send a message.

cat << EOF > msg.yaml
---
- toNodes:
    - node1
  method: cliCommand
  methodArgs:
    - "bash"
    - "-c"
    - |
      echo "some config line" > /etc/my-service-config.1
      echo "some config line" > /etc/my-service-config.2
      echo "some config line" > /etc/my-service-config.3
      systemctl restart my-service

  replyMethod: none
  ACKTimeout: 0
EOF

cp msg.yaml readfolder

With the above message we send to ourselves since we only got 1 node running. To start up more nodes repeat the above steps, replace the node name under toNodes with new names for new nodes. NB: If more nodes share the same name the requests will be loadbalanced between them round robin.

Install on a host

Start up a local nats message broker if you don't already have one.

docker run -p 4444:4444 nats -p 4444

Build the ctrl binary from the source code.

git clone git@github.com:postmannen/ctrl.git
cd cmd/ctrl
go run build

Copy the binary to /usr/local.

mkdir -p /usr/local/ctrl
cp ./ctrl /usr/local/ctrl

For testing we create a folder for the node to store it's data.

```bash
cd /usr/local/ctrl
mkdir node1
cd node1

ctrl will create all the folders needed like etc, var and more in the current directory where it was started if they don't already exist. This behaviour can be changed with flags or env variables.

Create a .env file for the startup options. Flags can also be used.

cat << EOF > .env
NODE_NAME="node1"
BROKER_ADDRESS="127.0.0,1:4444"
ENABLE_DEBUG=1
START_PUB_HELLO=60
IS_CENTRAL_ERROR_LOGGER=0
EOF

Start up ctrl. ctrl will automatically used the local .env file we created.

../usr/local/ctrl/ctrl

If you open another window, and go to the /usr/local/ctrl/node1 you should see that ctrl have created the directory structure for you with ./etc, ./var, ./directoryfolder and so on.

Prepare and send a message. We send messages by copying them into the ./readfolder where ctrl automatically will pick it up, and process it.

cat << EOF > msg.yaml
---
- toNodes:
    - node1
  method: cliCommand
  methodArgs:
    - "bash"
    - "-c"
    - |
      echo "some config line" > /etc/my-service-config.1
      echo "some config line" > /etc/my-service-config.2
      echo "some config line" > /etc/my-service-config.3
      systemctl restart my-service

  replyMethod: none
  ACKTimeout: 0
EOF

cp msg.yaml readfolder

With the above message we send to ourselves since we only got 1 node running. To start up more nodes repeat the above steps, replace.

Run as service

Create a systemctl unit file to run ctrl as a service on the host

progName="ctrl"
systemctlFile=/etc/systemd/system/$progName.service

cat >$systemctlFile <<EOF
[Unit]
Description=http->${progName} service
Documentation=https://github.com/postmannen/ctrl
After=network-online.target nss-lookup.target
Requires=network-online.target nss-lookup.target

[Service]
ExecStart=env CONFIG_FOLDER=/usr/local/${progName}/etc /usr/local/${progName}/${progName}

[Install]
WantedBy=multi-user.target
EOF

systemctl enable $progName.service &&
systemctl start $progName.service

NATS Server install

ctrl uses NATS as the messagaging backbone. The following text will describe how to quickly get up and running with a minimal NATS setup. For full details of what you can do with nats-server, check out the official docs at https://docs.nats.io/running-a-nats-service/introduction/installation

NKEY

NATS uses ED25519 based keys called NKEY's for Authentication and Authorization. The keys are created by a tool called nk. The instructions for how to install it are found here https://docs.nats.io/using-nats/nats-tools/nk.

The private key are called seed, and the public key are called user.

To create the keys run the following command after the nk tool is installed.

nk -gen user -pubout

The tool will print out two new keys. Where the private Seed starts with the letter S, and the public User key starts with the letter U.

The private Seed key are used with each ctrl instance, and are referenced as an ENV, flag, or via file.

The public User key are used in the nats-server config file for Authentication, to define access lists for what Nats Subjects the ctrl instances should be allowed to send to, or receive from.

Install the NATS Server

For this example we use docker compose to start the NATS server.

On your local computer create a folder to hold the NATS docker compose, and configuration files.

mkdir nats && cd nats

create the docker compose file called nats.yaml, with the following content.

version: "3"

services:
  nats:
    build: .
    image: nats:latest
    # -js enables jetstram on the nats server.
    command: "-c /app/nats-server.conf -D -js"
    restart: always
    ports:
      - "4222:4222"
    volumes:
      - ./nats.conf:/app/nats-server.conf
    logging:
        driver: "json-file"
        options:
            max-size: "10m"
            max-file: "10"

In the same directory create the nats-server.conf file, with the following content. Replace the placeholders for the user keys in the acl with the user keys you created earlier.

port: 4222

ACL = {
    publish: {
            allow: [">"]
    }
    subscribe: {
            allow: [">"]
    }
}

authorization: {
    timeout: "30s"
    users = [
        {
            # github
            nkey: <REPLACE WITH github user key here>
            permissions: $ACL
        },
        {
            # node1
            nkey: <REPLACE WITH seed user key here>
            permissions: $ACL
        },
    ]
}

Firewall openings for NATS Server

You will need to open your firewall for inbound tcp/4222 from the internet.

You can find your public ip address here https://ipv4.jsonip.com/.

Other

More details like how to use certificates to encrypt the communication can be found in the official nats docs, https://docs.nats.io/.

Core overview

This is just a quick introduction to the core concepts of ctrl. Details can be found by reading further in the Core ctrl section.

Concept

In ctrl all nodes are treated equally, and there is no concept of client/server. There are just nodes that can send messages to other nodes, and the messages contains commands of what to do on the node that receives the message.

A message can be sent from a node to one, many, groups or all nodes with a command to execute, and the result of when the command is done will be sent back to where the message originated.

Central

There should be at least one node acting as the central in each environment. The central node will have functionality like:

• Audit logging of all commands sent and executed on nodes.
• Handling and distributing keys for signing messages.
• Handling and distributing Access Lists (ACL) for authorizing messages.
• General error logging.

Messaging

The handling of all messages is done by spawning up a process for handling the message in it's own thread. This allows us for indivudual control of each message both in regards to ACK's, error handling, send retries, and reruns of methods for a message if the first run was not successful.

All messages processed by a publisher will be written to a log file after they are processed, with all the information needed to recreate the same message if needed, or it can be used for auditing.

Message types of both Acknowledge and No-Acknoweldge:

• ACKTimeout set to 0 will make the message become a No ACK message.
• ACKTimeout set to >=1 will make the message become an ACK message.

If ACK is expected, and not received, then the message will be retried the amount of time defined in the retries field of a message. If numer of retries are reached, and still no ACK received, the message will be discared, and a log message will be sent to the log server.

To make things easier, all timeouts used for messages can be set with env variables or flags at startup of ctrl. Locally specified timeouts directly in a message will override the startup values, and can be used for more granular control when needed.

Example of message flow:

Handling the result of a successful message delivery

Default reply

We can decide what to do with the result of a message, and it's method have run successfully. By default if nothing is specified ctrls file method will be used. The result will be written to ctrl's data folder, and structured with the name of the node the result came from. This behavior can be overridden by defining the directory and fileName fields in the message body.

If no reply is wanted, set the replyMethod to none

Setting your own replyMethod

All methods that can be used as a method in message, can also be used as a replyMethod.

As an example we can use the cliCommand also as replyMethod.

[
    {
        "toNode": "node2",
        "method": "cliCommand",
        "methodArgs": [
            "bash",
            "-c",
            "curl localhost:2111/metrics"
        ],
        "replyMethod": "cliCommand",
        "replyMethodArgs": [
            "bash",
            "-c",
            "echo \"{{CTRL_DATA}}\"> /somedirectory/somefile.html"
        ],
        "methodTimeout": 10
    }
]

The use of {{CTRL_DATA}} allows us to take the result of the initial method, and embed that in the reply message. The use are further explained in {{CTRL_DATA}} variable

Message fields

Schema for the message structure to use with ctrl

More detailed description of the fields

toNode : "some-node"

The node to send the message to.

toNodes : node1,node2

ToNodes to specify several hosts to send message to in the form of an slice/array. When used, the message will be split up into one message for each node specified, so the sending to each node will be handled individually.

data : data here in byte format

The actual data in the message. This is the field where we put the returned data in a reply message. The data field are of type []byte.

method : cliCommand

What request method type to use, like cliCommand, httpGet, all methods.

  methodArgs :
  - "bash"
  - "-c"
  - |
      echo "this is a test"
      echo "and some other test"

Additional arguments that might be needed when executing the method. Can be f.ex. an ip address if it is a tcp sender, or the actual shell command to execute in a cli.

replyMethod : file

ReplyMethod, is the method to use for the reply message. By default the reply method will be set to log to file, but you can override it setting your own here.

  methodArgs :
  - "bash"
  - "-c"
  - |
      echo "this is a test"

Additional arguments that might be needed when executing the reply method. Can be f.ex. an ip address if it is a tcp sender, or the shell command to execute in a cli session. replyMethodArgs :

fromNode Node : "node2"

From what node the message originated. This field is automatically filled by ctrl when left blanc, so that when a message are sent from a node the user don't have to worry about getting eventual repply messages back. Setting a

ACKTimeout: 10 

ACKTimeout for waiting for an Ack message in seconds.

If the ACKTimeout value is set to 0 the message will become an No Ack message. With No Ack messages we will not wait for an Ack, nor will the receiver send an Ack, and we will never try to resend a message.

retryWait : 60

RetryWait specifies the time in seconds to wait between retries. This value is added to the ACKTimeout and to make the total time before retrying to sending a message. A usecase can be when you want a low ACKTimeout, but you want to add more time between the retries to avoid spamming the receiving node.

retries : 3

How many times we should try to resend the message if no ACK was received.

replyACKTimeout int `json:"replyACKTimeout" yaml:"replyACKTimeout"`

The ACK timeout used with reply messages. If not specified the value of ACKTimeout will be used.

replyRetries int `json:"replyRetries" yaml:"replyRetries"`

The number of retries for trying to resend a message for reply messages.

methodTimeout : 10

Timeout for how long a method should be allowed to run before it is timed out. If methodTimeout : -1 the method will not time out.

replyMethodTimeout : 10

Timeout for how long a method should be allowed to run before it is timed out, but for the reply message.

directory string `json:"directory" yaml:"directory"`

Specify the directory structure to use when saving the result data for a repply message.

• When the values are comma separated like "syslog","metrics" a syslog folder with a subfolder metrics will be created in the directory specified with startup env variable SUBSCRIBER_DATA_FOLDER.
• Absolute paths can also be used if you want to save the result somewhere else on the system, like "/etc/myservice".

fileName : myfile.conf

The fileName field are used together with the directory field mentioned earlier to create a full path for where to write the resulting data of a repply message.

schedule : [2,5]

Schedule are used for scheduling the method of messages to be executed several times. The schedule is defined as an array of two values, where the first value defines how often the schedule should trigger a run in seconds, and the second value is for how long the schedule are allowed to run in seconds total. [2,5] will trigger the method to be executed again every 2 seconds, but when a total time of 5 seconds have passed it will stop scheduling.

Jetstream

ctrl takes benefit of some of the streaming features of NATS Jetstream. In short, Jetstream will keep the state of the message queues on the nats-server, whereas with normal ctrl messaging the state of all the messages are kept on and is the responsibility of the node publishing the message.

With Jetstream some cool posibilities with ctrl become possible. A couple of examples are:

• Use ctrl in a github runner, and make messages available to be consumed even after the runner have stopped.
• Broadcast message to all nodes.

General use

To use Jetstream instead of regular NATS messaging, put the node name of the node that is supposed to consume the message in the jetstreamToNode field.

---
- toNodes:
    - mynode1
  jetstreamToNode: mynode1
  method: cliCommand
  methodArgs:
    - /bin/bash
    - -c
    - |
      tree
  methodTimeout: 3
  replyMethod: console
  ACKTimeout: 0

The request will then be published with NATS Jetstream to all nodes registered on the nats-server. The reply with the result will be sent back as a normal NATS message (not Jetstream).

Broadcast

A message can be broadcasted to all nodes by using the value all with jetstreamToNode field of the message like the example below.

---
- toNodes:
    - all
  jetstreamToNode: all
  method: cliCommand
  methodArgs:
    - /bin/bash
    - -c
    - |
      tree
  methodTimeout: 3
  replyMethod: console
  ACKTimeout: 0

The request will then be published with NATS Jetstream to all nodes registered on the nats-server. The reply with the result will be sent back as a normal NATS message (not Jetstream).

Specify more subjects to consume with streams

More subject can be specified by using the flag jetstreamsConsume or env variable JETSTREAMS_CONSUME .

Example:

JETSTREAMS_CONSUME=updates,restart,indreostfold

Request Methods

{{CTRL_DATA}} variable

By using the {{CTRL_DATA}} you can grab the result output of your initial request method, and then use it as input in your reply method.

NB: The echo command in the example below will remove new lines from the data. To also keep any new lines we need to put escaped quotes around the template variable. Like this:

• \"{{CTRL_DATA}}\"

Example of usage:

[
    {
        "directory":"cli_command_test",
        "fileName":"cli_command.result",
        "toNode": "node2",
        "method":"cliCommand",
        "methodArgs": ["bash","-c","tree"],
        "replyMethod":"cliCommand",
        "replyMethodArgs": ["bash", "-c","echo \"{{CTRL_DATA}}\" > apekatt.txt"],
        "replyMethodTimeOut": 10,
        "ACKTimeout":3,
        "retries":3,
        "methodTimeout": 10
    }
]

The above example, with steps explained:

• Send a message from node1 to node2 with a Request Method of type cliCommand.
• When received at node2 we execute the Reqest Method with the arguments specified in the methodArgs.
• When the method on node2 is done the result data of the method run will be stored in the variable {{CTRL_DATA}}. We can then use this variable when we craft the reply message method by embedding it into a new bash command.
• The reply message is then sent back to node1, the method will be executed, and all newlines in the result data will be removed, and all the data with the new lines removed will be stored in a file called apekatt.txt

The same using bash's herestring:

[
    {
        "directory":"cli_command_test",
        "fileName":"cli_command.result",
        "toNode": "ship2",
        "method":"cliCommand",
        "methodArgs": ["bash","-c","tree"],
        "replyMethod":"cliCommand",
        "replyMethodArgs": ["bash", "-c","cat <<< {{CTRL_DATA}} > hest.txt"],
        "replyMethodTimeOut": 10,
        "ACKTimeout":3,
        "retries":3,
        "methodTimeout": 10
    }
]

Message methodArgs variables

{{CTRL_FILE}}

Read a local text file, and embed the content of the file into the methodArgs.

---
- toNodes:
    - btdev1
  #jetstreamToNode: btdev1
  method: cliCommand
  methodArgs:
    - /bin/bash
    - -c
    - |
      echo {{CTRL_FILE:/some_directory/source_file.yaml}}>/other_directory/destination_file.yaml
  methodTimeout: 3
  replyMethod: console
  ACKTimeout: 0

The above example will before sending the message read the content of the file /some_directory/source_file.yaml. When the message is received at it's destination node and the cliCommand is executed the content will be written to /other_directory/destination_file.yaml.

Nats timeouts

The various timeouts for the NATS messages can be controlled via an .env file or flags.

If the network media is a high latency like satellite links, it will make sense to adjust the client timeout to reflect the latency

  -natsConnOptTimeout int
        default nats client conn timeout in seconds (default 20)

The interval in seconds the nats client should try to reconnect to the nats-server if the connection is lost.

  -natsConnectRetryInterval int
        default nats retry connect interval in seconds. (default 10)

Jitter values.

  -natsReconnectJitter int
        default nats ReconnectJitter interval in milliseconds. (default 100)
  -natsReconnectJitterTLS int
        default nats ReconnectJitterTLS interval in seconds. (default 5)

Startup folder

Messages can be automatically scheduled to be read and executed at startup of ctrl.

A folder named startup will be present in the working directory of ctrl. To inject messages at startup, put them here.

Messages put in the startup folder will not be sent to the broker but handled locally, and only (eventually) the reply message from the Request Method called will be sent to the broker.

This can be really handy when you have some 1-off job you want to done at startup, like some cleaning up.

Example reqest metrics from nodes

Another example could be that you have some local prometheus metrics you want to scrape every 5 minutes, and you want to send them to some central metrics system.

---
- toNodes:
    - ["node1","node2"]
  method: httpGet
  methodArgs:
    - "http://localhost:8080/metrics"
  replyMethod: file
  methodTimeout: 5
  directory: metrics
  fileName: metrics.html
  schedule : [120,999999999]

The example above will send out a request to node1 and node2 every 120 second to scrape the metrics and write the results that came back to a folder named data/metrics in the current running directory.

Example read metrics locally first, and then send to remote node

But we can also make the nodes publish their metrics instead of requesting it by putting a message in each nodes startup folder, set the toNode field to local, and instead use the fromNode field to decide where to deliver the result.

---
- toNodes:
    - ["local"]
  fromNode: my-metrics-node
  method: httpGet
  methodArgs:
    - "http://localhost:8080/metrics"
  replyMethod: file
  methodTimeout: 5
  directory: metrics
  fileName: metrics.html
  schedule : [120,999999999]

In the above example, the httpGet will be run on the local node, and the result will be sent to my-metrics-node.

How to send the reply to another node further explained

Normally the fromNode field is automatically filled in with the node name of the node where a message originated. Since messages within the startup folder is not received from another node via the normal message path we need set the fromNode field in the message to where we want the reply (result) delivered.

As an example. If You want to place a message on the startup folder of node1 and send the result to node2. Specify node2 as the fromNode, and node1 as the toNode

Use local as the toNode nodename

Since messages used in startup folder are ment to be delivered locally we can simplify things a bit by setting the toNode field value of the message to local.

[
    {
        "toNode": "local",
        "fromNode": "central",
        "method": "cliCommand",
        "methodArgs": [
            "bash",
            "-c",
            "curl localhost:2111/metrics"
        ],
        "replyMethod": "console",
        "methodTimeout": 10
    }
]

This example message will be read at startup, and executed on the local node where it was read, the method will be executed, and the result of the method will be sent to central. This is basically the same as the previous example, but we're using cliCommand method with curl instead of the httpGet method.

Errors

Errors happening on all nodes will be reported back to the node(s) started with the flag -isCentralErrorLogger set to true, or by using the IS_CENTRAL_ERROR_LOGGER env variable.

Log level

The log level can also be specified with the LOG_LEVEL env variable. Values are error/info/warning/debug/none.

Debug

To get more debug information in the logs printed to STDERR the env variable ENABLE_DEBUG can be set to true. This will not affect the information printed to the log files, only to STDERR.

central

To get functionality like central audit log, signing keys, authorization with ACL's and hello messages one node should be started with the node name central

Hello messages

All nodes can send hello messages to inform that they are up. The interval between sending a hello message can be set with the START_PUB_HELLO environment variable.

Hello messages are sent to the node with the name central. When a hello message are received on central, information with the time and node name will be stored in the ctrl data folder

Public keys

ctrl nodes can use ed25519 keys for signing messages, so each ctrl instance will generate a public and private key pair on startup. The public keys are sent to the central server with the hello messages.

To read more about signing keys here: signing keys

signing keys

ACL

audit log

WebUI Overview

The WebUI is a web application that allows you to interact with CTRL via NATS.

NB: The WebUI is in early development.

Features

Send Commands

Send commands to one or many nodes. The command will be sendt, executed, and the result will be displayed in the WebUI.

Network Graph

Visualize the network of nodes.

The graph shows the nodes that are connected to the central node, and the time since the last hello message was received from the node.

When hovering over a node, the node details will be shown in a tooltip. The defined file templates for the node will also be shown in the tooltip, and the user can select one of the templates to use for the command to execute on the node.

File Templates

File templates containing a script can be used to execute commands. The file templates should be stored in the files directory on the central node. The WebUI will then ask the central node for available templates, and show the file templates in a dropdown menu for the user to select from.

Settings

Set setting like the Node name of the UI, NATS server URL, and NKEY to use for authentication to the NATS server.

Flame Graph

In development.

httpGet

In JSON.

[
    {
        "directory": "httpget",
        "fileName": "finn.no.html",
        "toNodes": ["node1","node2"],
        "method":"httpGet",
        "methodArgs": ["https://finn.no"],
        "replyMethod":"file",
        "ACKTimeout":5,
        "retries":3,
        "methodTimeout": 5
    }
]

In YAML.

---
- toNodes:
    - ["node1","node2"]
  method: httpGet
  methodArgs:
    - "https://finn.no"
  replyMethod: file
  ACKTimeout: 5
  retries: 3
  methodTimeout: 5
  directory: httpget
  fileName: finn.no.html

The result html file of the http get will be written to:

• <data folder>\httpget\node1\finn.no.html
• <data folder>\httpget\node2\finn.no.html

tailFile

In JSON.

[
    {
        "directory": "tails",
        "fileName": "some.log",
        "toNodes": "node1","node2","node3",
        "method":"tailFile",
        "methodArgs": ["/var/log/syslog"],
        "ACKTimeout":5,
        "retries":3,
        "methodTimeout": 200
    }
]

NB: If no replyMethod are specified, it will default to file

In YAML.

---
- toNodes:
    - ["node1","node2","node3"]
  method: tailFile
  methodArgs:
    - "/var/log/syslog"
  replyMethod: file
  ACKTimeout: 5
  retries: 3
  methodTimeout: 5
  directory: tails
  fileName: var_log_syslog.log

The above example will tail the syslog file on 3 nodes for 5 seconds, and save the result on the node where the request came from in the local data folder.

cliCommand

With cliCommand you specify the command to run in methodArgs prefixed with the interpreter to use, for example with bash bash "bash","-c","tree".

On Linux and Darwin, the shell interpreter can also be auto detected by setting the value of useDetectedShell in the message to true. If set to true the methodArgs only need a single string value with command to run. Example below.

---
- toNodes:
    - node2
  useDetectedShell: true
  method: cliCommand
  methodArgs:
    - |
      rm -rf ./data & systemctl restart ctrl

  replyMethod: fileAppend
  ACKTimeout: 30
  retries: 1
  ACKTimeout: 30
  directory: system
  fileName: system.log

Example in JSON

[
    {
        "directory":"system",
        "fileName":"system.log",
        "toNodes": ["node2"],
        "method":"cliCommand",
        "methodArgs": ["bash","-c","rm -rf ./data & systemctl restart ctrl"],
        "replyMethod":"fileAppend",
        "ACKTimeout":30,
        "retries":1,
        "methodTimeout": 30
    }
]

Example in YAML

In YAML.

---
- toNodes:
    - node2
  method: cliCommand
  methodArgs:
    - "bash"
    - "-c"
    - |
      rm -rf ./data & systemctl restart ctrl

  replyMethod: fileAppend
  ACKTimeout: 30
  retries: 1
  ACKTimeout: 30
  directory: system
  fileName: system.log

Will send a message to node2 to delete the ctrl data folder, and then restart ctrl. The end result will be appended to the specified file on the node where the request originated.

More examples

Get the prometheus metrics of the central server

[
    {
        "toNode": "central",
        "method": "cliCommand",
        "methodArgs": [
            "bash",
            "-c",
            "curl localhost:2111/metrics"
        ],
        "replyMethod": "console",
        "methodTimeout": 10
    }
]

Start up a tcp listener for number of seconds

[
    {
        "toNode": "node1",
        "method": "cliCommandCont",
        "methodArgs": [
            "bash",
            "-c",
            "nc -lk localhost 8888"
        ],
        "replyMethod": "toConsole",
        "methodTimeout": 10,
    }
]

The netcat tcp listener will run for 10 seconds before the method timeout kicks in and ends the process.

Get the running docker containers from a node

[
    {
        "directory":"some/cli/command",
        "fileName":"cli.result",
        "toNode": "node2",
        "method":"cliCommand",
        "methodArgs": ["bash","-c","docker ps -a"],
        "replyMethod":"fileAppend",
    }
]

cliCommandCont

The cliCommand and the cliCommandCont are the same, except for one thing. cliCommand will wait until wether the method is finished or the methodTimeout kicks in to send the result as one single message. cliCommand will when a line is given to either stdout or stderr create messages with that single line in the data field, and send it back to the node where the message originated.

[
    {
        "directory":"some/cli/command",
        "fileName":"cli.result",
        "toNode": "node2",
        "method":"cliCommandCont",
        "methodArgs": ["bash","-c","tcpdump -nni any port 8080"],
        "replyMethod":"fileAppend",
        "methodTimeout":10,
    }
]

Example Will run the command given for 10 seconds (methodTimeout), and return the stdout output of the command continously while the command runs. Uses the methodTimeout to define for how long the command will run.

copySrc

Copy a file from one node to another node.

[
    {
        "directory": "copy",
        "fileName": "copy.log",
        "toNodes": ["central"],
        "method":"copySrc",
        "methodArgs": ["./testbinary","ship1","./testbinary-copied","500000","20","0770"],
        "methodTimeout": 10,
        "replyMethod":"console"
    }
]

• toNode/toNodes, specifies what node to send the request to, and which also contains the src file to copy.
• methodArgs, are split into several fields, where each field specifies:
  1. SrcFullPath, specifies the full path including the name of the file to copy.
  2. DstNode, the destination node to copy the file to.
  3. DstFullPath, the full path including the name of the destination file. The filename can be different than the original name.
  4. SplitChunkSize, the size of the chunks to split the file into for transfer.
  5. MaxTotalCopyTime, specifies the maximum allowed time the complete copy should take. Make sure you set this long enough to allow the transfer to complete.
  6. FolderPermission, the permissions to set on the destination folder if it does not exist and needs to be created. Will default to 0755 if no value is set.

To copy from a remote node to the local node, you specify the remote nodeName in the toNode field, and the message will be forwarded to the remote node. The copying request will then be picked up by the remote node's copySrc handler, and the copy session will then be handled from the remote node.

Send more messages

[
    {
        "directory":"cli-command-executed-result",
        "fileName": "some.log",
        "toNode": "ship1",
        "method":"cliCommand",
        "methodArgs": ["bash","-c","sleep 3 & tree ./"],
        "ACKTimeout":10,
        "retries":3,
        "methodTimeout": 4
    },
    {
        "directory":"cli-command-executed-result",
        "fileName": "some.log",
        "toNode": "ship2",
        "method":"cliCommand",
        "methodArgs": ["bash","-c","sleep 3 & tree ./"],
        "ACKTimeout":10,
        "retries":3,
        "methodTimeout": 4
    }
]

ctrl as github action runner

Run ctrl as a docker container in a github workflow. This can for example be as part of a CI/CD pipeline, or for content versioning.

Or with other words.. you have a github repository that holds the instructions for what ctrl should do. There is a github action attached to it. When the repository is updated, the github action will start a ctrl container (runner) which will read the instructions you pushed to the repository. The runner will then effectuate to instructions on all the nodes and the commands defined in the instructions.

This howto assumes that you have a nats-server setup, and at least one ctrl node instance up running. How to setup a basic nats-server and a ctrl node on a computer/server/container can be found in the User Guides section of the documentation.

In the examples below I've used the name node1 as an example for the node that will receive the message when the github repository are updated, but this could be any number of nodes you'd like.

Github Action Runner setup

Create a Github repository.

Clone the repository down to your local machine or other.

git clone <my-repo-name> && cd my-repo-name

Create a github workflows folder

mkdir -p .github/workflows

Create a workflow.yaml in the new directory with the following content.

name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
on: [push]

jobs:
  send-message:
    name: send-message
    runs-on: ubuntu-latest

    services:
      ctrl:
        image: postmannen/ctrl:amd64-0.03
        env:
          NKEY_SEED: ${{ secrets.SEED }}
          NODE_NAME: "github"
          BROKER_ADDRESS: "<REPLACE WITH ip address of the NATS broker here>:4222"
          ENABLE_KEY_UPDATES: 0
          ENABLE_ACL_UPDATES: 0
        volumes:
          # mount the readfolder where we put the message to send.
          - ${{ github.workspace }}/readfolder:/app/readfolder
          # mount the files folder from the repo to get the file
          # to inject into the message.
          - ${{ github.workspace }}/files:/app/files
        options: --name ctrl

    steps:
      # When the container was started the repo was not yeat available since it was not
      # the workspace at that time. We want to make the /files mount to point to the
      # folder in the repo. Checkout the git repo, and restart the docker container so
      # it will mount the directories from the repo which now is the current github.workspace.
      - name: checkout repo
        uses: actions/checkout@v4
      - name: restart docker
        uses: docker://docker
        with:
          args: docker restart ctrl
      - run: sudo chmod 777 ${{ github.workspace }}/readfolder
      - run: sleep 3
      # Send the message by moving it to the readfolder.
      - run: mv ${{ github.workspace }}/message.yaml ${{ github.workspace }}/readfolder
      - run: >
          sleep 5 && tree

Define NKEY as github secret

In your repository, go to Settings->Secrets And Variables->Actions and press New Repository Secret. Give the new secret the name SEED, and put the content of the seed into Secret. This is the seed that we referenced earlier in the github action workflow.

Define message with command to send

We want to send a message from the Github Action, so we need to specify the content of the message to use.

In the root folder of the github folder create a message.yaml file, and fill in the following content:

---
- toNodes:
    - node1
  method: cliCommand
  methodArgs:
    - "bash" # Change from bash to ash for alpine containers
    - "-c"
    - |
      echo '{{CTRL_FILE:./files/file.txt}}'>file.txt

  replyMethod: none
  ACKTimeout: 0

The message references a file with the {{CTRL_FILE:<file path>}} to use with the cli command found in the files folder. The file referenced will be embedded into the methodArgs defined in the message.

From the repository folder run the following commands:

mkdir -p files
echo "some cool text to put as file content.........." > files/file.txt

The example tries to show how we can get the message to run a shell/cli command on node1 at delivery. The shell command will use the content of the file located at <repo>/files/file.txt, and create a file called file.txt in the ctrl working directory on node1.

Update the repository and send message with command

Commit the changes of the repository. If you check the Actions section of the new repo you should see that a an action have started.

When the action is done, you should have received a file called file.txt in the ctrl working directory on node1, with the content you provided in text.txt.

Other cool things you can do ... like deploy kubernetes manifests

Replace the the bash command specified in the method arguments with a kubetctl command like this:

---
- toNodes:
    - node1
  method: cliCommand
  methodArgs:
    - "bash" # Change from bash to ash for alpine containers
    - "-c"
    - |
      kubectl apply -f '{{CTRL_FILE:./files/mydeployment.yaml}}'

  replyMethod: none
  ACKTimeout: 0

Create a new file in the <repo>/files folder named mydeployment.yaml.

Commit the changes to the repo, and the deployment should be executed if you have a kubernetes instance running on node1.

ctrl as prometheus collector

ctrl can be used to collect collect metrics from various systems. It can for example scrape/read some defined metrics with one node, and deliver the result to another node where you can expose the metrics directly using ctrl's builtin http server, or you could for example inject the metrics data to some database if that is desired. It is totally up to you.

The example that follows will scrape prometheus metrics with two ctrl nodes, deliver them to a third node called metrics, and expose them with the builtin http server. Prometheus can then be used to read all the metrics from the various nodes on the metrics node.

Before you start, make sure to read the User Guides section for how to start up a NATS broker, and for general information about setting up ctrl.

Node Setup

central node

Start up a ctrl node named central that will serve as central for audit logs and other system logs happening with the ctrl nodes. Example of .env file to use.

NODE_NAME=central
BROKER_ADDRESS=localhost:4222
LOG_LEVEL=info
START_PUB_HELLO=60
IS_CENTRAL_ERROR_LOGGER=true

collected metrics node

Start up a node named metrics that will serve as the central place where we deliver all the metrics. On this node we expose ctrl's data folder over http. Example .env file below.

NODE_NAME=metrics
BROKER_ADDRESS=localhost:4222
LOG_LEVEL=info
START_PUB_HELLO=60
EXPOSE_DATA_FOLDER=localhost:6060

metrics collector nodes

prometheus node exporter

If don't yet have any metrics to collect, you can start up prometheus node_exporter to get some local system metrics.

collector node setup

The following configuration file can be used on all nodes, but for each node started replace with a unique NODE_NAME, eg. node1 and node2.

NODE_NAME=node1     #give each node a unique node name
BROKER_ADDRESS=localhost:4222
LOG_LEVEL=info
START_PUB_HELLO=60

Create 2-3 (node1,node2,node3) nodes which will be the nodes that scrape some metrics on some host.

Startup folder

Each ctrl instance started will have a startup folder in it's running directory. Messages in the startup folder will be read at startup, and handled by ctrl.

We can use the schedule field in the message to make ctrl rerun the method of the message at a scheduled interval.

Put the following message in the startup folder on all the nodes that will collect metrics.

---
- toNodes:
    # Deliver the message locally
    - local
  # Set the node where we send the reply with the result
  fromNode: metrics
  method: httpGet
  methodArgs:
    - http://localhost:9100/metrics
  # Write the result to a file on the fromNode
  replyMethod: file
  # The directory which we want to write the result in
  directory: nodeexporters
  # The filename we want to write the result to
  fileName: metrics.html
  # Schedule rerun of the method every 30 second, for 999999999 seconds.
  schedule: [30,999999999]

Check the collected metrics

When all nodes are started, they should start to send metrics to the metrics node. We can see the result by using curl on the node named metrics.

http://localhost:6060/nodeexporters/node1/metrics.html
http://localhost:6060/nodeexporters/node2/metrics.html

If you want to do something further with this example you can install prometheus on the metrics node, and direct it to collect read the metrics from the various folders via the url's above.

ctrl as tcp forwarder for ssh

ctrl can be used to forward a TCP stream between two nodes. The Individual TCP packet will be read from a TCP listener on a node, put into a standard ctrl message, the message are then sent to the destination node using NATS PUB/SUB where the TCP packet is extracted, and written to the TCP connection.

In the follwing example we have two nodes, where we can think of node1 as the local node running on your computer, and node2 are the remote node running on a server somewhere.

We want to connect with ssh to node2, but it is not directly available from the network where the local node1 resides, so we use ctrl as a proxy server and forward the ssh tcp stream.

Steps

1. Create the message with forwarding details, and copy iy into node1's readfolder.

---
- toNodes:
    - node1                 # The source node where we start the tcp listener
  method: portSrc           # The ctrl tcp forward method
  methodArgs:
    - node2                 # The destination node who connects to the actual endpoint
    - 192.168.99.1:22       # The ip address and port we want to connect to from endpoint.
    - :10022                # The local port to start the listener on node1
    - 30                    # How many seconds the tcp forwarder should be active
  methodTimeout: 30         # Same as above, but at the method level. Set them to the same.
  replyMethod: console      # Do logging to console on node1
  ACKTimeout: 0             # No ACK'in of messages. Fire and forget.

1. On node1 a process with a TCP listener will be started at 0.0.0.0:10022.

2. The process on node1 will then send a message to node2 to start up a process and connect to the endpoint defined in the methodArgs.

3. The forwarding are now up running, and we can use a ssh client to connect to the endpoint which are now forwarded the node1 on port 10022.

ssh -p 10022 user@localhost

The forwarding will automatically end after the timeperiod specified, which in this example is 30 seconds.