Introduction

This guide offers troubleshooting procedures for an appliance that has become offline or failed.  

Offline appliances can stem from a variety of issues: hardware malfunction, YubiKey, VPN, security device blocking the appliance, the firewall rules, etc. The point of this guide is to understand how each issue can show the appliance as offline and the appropriate order of steps to follow.

If the appliance fails to boot entirely, skip straight to replacing the appliance as troubleshooting is greatly hindered.

What happens when an appliance goes offline?

• The network traffic will no longer be monitored.
• For a primary appliance, the endpoint agents will not be able to check in. They will still collect data but no alerts for endpoints would get created. Once the appliance is back online, it will clear the backlog of data from the agents, but with the primary appliance offline, the client is not receiving alerts in real time.
• The only service not affected is their cloud monitoring; AROs would still get produced for those.

Troubleshooting an offline appliance:

When troubleshooting hardware, it is always wise to start with the basics; ensure the power and other cables are properly connected as per your appliance's configuration guide. Additionally, you may be impacted by a recent change in your environment that did not consider the appliance. 

1: Any recent changes or issues?

Since the appliance was set up and working for a certain amount of time prior to going offline, this is the best place to start. There is a high chance that something changed for the appliance to now show as offline, so check with your team members if any recent changes were made.

Examples of recent changes or issues that may impact the appliance's connectivity: 

• A new security device (firewall) was installed. 
• The appliance was moved in the environment.
• New configurations, setups, or other changes with the client’s network, firewall, etc. 
• Power outage.

If a change or issue did recently occur, skip to the most appropriate troubleshooting step below. For example, if you experienced a power outage, then navigate to "2: Can you ping the appliance locally?". If you replaced or made any changes to your firewall(s), then skip to "4: Firewall Rules and Logs".

2: Can you ping the appliance locally?

To verify that it's getting a local IP address, please check the following:

1. Ping the appliance from your network.
2. Ensure the appliance is receiving a local IP from your switch.
3. If reachable by network, verify the connectivity using a self-check method such as:
   1. Connect to the appliance using SSH and open the Appliance Management Console 
   2. Browse to the Field Effect Portal's Status Page and verify the appliance's connectivity status.

This should be a relatively quick check and could point right at the potential issue. If you are able to ping the appliance's IP address locally (from the same network as the appliance), then that directly points to an issue with the VPN. This would prove that the appliance is receiving an IP and is connected to the internet, but Field Effect MDR is unable to connect to the appliance. The VPN connection being blocked points to a misconfigured firewall rule to enable Field Effect VPN access, so skip to Step 4.

If you are unable to ping the appliance’s IP, that would mean the appliance is most likely not receiving an IP address, and you need to troubleshoot the local network. 

The appliance's green port is configured to pick up a DHCP IP from your network, which will assign the appliance a dynamic IP address. If you have configured the appliance to use a static IP, the red port would be used for Field Effect to connect to the appliance (with the exception of Oskar devices, where the green and red port are one and the same). 

Please check that the cabling is properly connected for use case; DHCP or static IP.

Step 3: Check the hardware

If you experience a power outage or other similar circumstance, start with rebooting the device. The below order is in a logical sequence if physical changes were made to the appliance.

Here are some common workflows you can use to check hardware issues with the appliance: 

3.1: Check the YubiKey connection

The YubiKey is a form of physical security that stops the appliance from booting if it is not connected. So, make sure it’s plugged in snugly and in the correct orientation:

3.2: Check the cable connections 

If your appliance is offline, it could simply be powered off or have a loose cable connection. Please check all the cables to they are tightly plugged in, and the appliance is receiving power. 

• Check network cables connecting the appliance.
• Check for power, and/or try a different power source.
• Ensure that the USB YubiKey (encryption module) is fully seated the port. 
  • Also try another USB port on the device if required.
• Reboot the appliance.
  • If the appliance does not come back online after a reboot, please connect a keyboard and monitor to the appliance.
• Check for a login prompt and login locally to the Appliance Management Console or browse to the Field Effect Portal's Status page.

To note, issues with the power cable and green cable could cause the appliance to go offline. The power cable, for obvious reasons, and the green cable as it provides internet connectivity to the appliance. The yellow and blue cables pass network traffic to the appliance and should not cause issues with connectivity.

3.3: Reboot the device 

After checking on the YubiKey and cables, in case any of them created the issue, the appliance still might still need a reboot to come back online. For example, if the green cable was loose, the appliance may need a reboot to acquire an IP again. 

A reboot should normally consist of turning off the appliance off for 30 seconds and then powering it back on. The power buttons are on the back for the B1, E1 and E100 – for the Oskar, you will need to unplug the power cable to reboot it.

4: Check firewall rules and logs

If you can ping the appliance's IP address locally, the appliance is online - but it is unable to communicate back with Field Effect MDR. If this is the case, ensure that our relay connection is allowed as an outbound connection within your firewall(s) rules. 

You can use the Appliance Status Page to find your appliance's relay server hostname, that needs to be allowed as an outbound connection.

Firewall rules must be set to ensure that the appliance, and endpoint agents, can function correctly. The OpenVPN rule, if misconfigured, could cause connectivity issues with the appliance.  

Example Scenario:

A firewall is blocking both UDP port 1194 (OpenVPN) and TCP port 443 (HTTPS) connectivity from your network to Field Effect datacenter.

This firewall rule should have already been set if the appliance was working previously, but if any changes were made to the firewall, it’s possible the rule was altered or removed.

Please refer to your firewall logs, as it would reveal if anything was being blocked – such as the required OpenVPN firewall rule. 

5: Check Appliance Logs

To check on the appliance logs, you will need to login to the Appliance Management Console at the appliance physically (with peripherals) or by an SSH connection. 

Feel free to do this independently, or by scheduling a call with the Field Effect support team. Please reach out to support@fieldeffect.com to get the credentials for your appliance, or to schedule a call to go through the logs.

6: Replace the Appliance

If all the remediation steps above were taken and no solution was found, contact Support to discuss replacing the appliance.