Introduction

If the sensor fails to boot entirely, skip straight to replacing the appliance as troubleshooting is greatly hindered.

This guide offers troubleshooting procedures for sensors that has become offline or failed.  

Offline sensors can stem from a variety of issues: hardware malfunction, YubiKey, VPN, security device blocking the sensor, the firewall rules, etc. The point of this guide is to understand how each issue can show the sensor as offline and the appropriate order of steps to follow.

This article covers the following:  

• What happens when an sensor goes offline?
• Troubleshooting an offline sensor:
  • 1: Any recent changes or issues?
  • 2: Can you ping the sensor locally?
  • Step 3: Check the hardware
    • 3.1: Check the YubiKey connection
    • 3.2: Check the cable connections 
    • 3.3: Reboot the device 
  • 4: Check firewall rules and logs
    • Example Scenario:
  • 5: Check sensor Logs
  • 6: Replace the sensor

What happens when an sensor goes offline?

• The network traffic will no longer be monitored.
• For an MDR SIEM sensor (formerly termed "primary appliance"), the endpoint agents will not be able to check in. They will still collect data but no alerts for endpoints would get created. Once it is back online, it will clear the backlog of data from the agents, but with the MDR SIEM sensor offline, the client is not receiving alerts in real time.
• The only service not affected is their cloud monitoring; AROs would still get produced for those.

Troubleshooting an offline sensor:

When troubleshooting hardware, it is always wise to start with the basics; ensure the power and other cables are properly connected as per your sensor's configuration guide. Additionally, you may be impacted by a recent change in your environment that did not consider the sensor. 

1: Any recent changes or issues?

Since the sensor was set up and working for a certain amount of time prior to going offline, this is the best place to start. There is a high chance that something changed for the sensor to now show as offline, so check with your team members if any recent changes were made.

Examples of recent changes or issues that may impact the sensor's connectivity: 

• A new security device (firewall) was installed. 
• The sensor was moved in the environment.
• New configurations, setups, or other changes with the client’s network, firewall, etc. 
• Power outage.

If a change or issue did recently occur, skip to the most sensor troubleshooting step below. For example, if you experienced a power outage, then navigate to "2: Can you ping the sensor locally?". If you replaced or made any changes to your firewall(s), then skip to "4: Firewall Rules and Logs".

2: Can you ping the sensor locally?

To verify that it's getting a local IP address, please check the following:

1. Ping the sensor from your network.
2. Ensure the sensor is receiving a local IP from your switch.
3. If reachable by network, verify the connectivity using a self-check method such as:
   1. Connect to the sensor using SSH and open the Management Console 
   2. Browse to the Field Effect Portal's Status Page and verify the sensor's connectivity status.

This should be a relatively quick check and could point right at the potential issue. If you are able to ping the sensor's IP address locally (from the same network as the sensor), then that directly points to an issue with the VPN. This would prove that the sensor is receiving an IP and is connected to the internet, but Field Effect MDR is unable to connect to it. The VPN connection being blocked points to a misconfigured firewall rule to enable Field Effect VPN access, so skip to Step 4.

If you are unable to ping the sensor’s IP, that would mean the sensor is most likely not receiving an IP address, and you need to troubleshoot the local network. 

The sensor's green port is configured to pick up a DHCP IP from your network, which will assign the sensor a dynamic IP address. If you have configured the sensor to use a static IP, the red port would be used for Field Effect to connect to the sensor (with the exception of Oskar devices, where the green and red port are one and the same). 

Please check that the cabling is properly connected for use case; DHCP or static IP.

Step 3: Check the hardware

If you experience a power outage or other similar circumstance, start with rebooting the device. The below order is in a logical sequence if physical changes were made to it.

Here are some common workflows you can use to check hardware issues with the sensor: 

3.1: Check the YubiKey connection

The YubiKey is a form of physical security that stops the sensor from booting if it is not connected. So, make sure it’s plugged in snugly and in the correct orientation:

3.2: Check the cable connections 

If your sensor is offline, it could simply be powered off or have a loose cable connection. Please check all the cables to they are tightly plugged in, and it is receiving power. 

• Check network cables connecting the sensor.
• Check for power, and/or try a different power source.
• Ensure that the USB YubiKey (encryption module) is fully seated the port. 
  • Also try another USB port on the device if required.
• Reboot the sensor.
  • If the sensor does not come back online after a reboot, please connect a keyboard and monitor to the sensor.
• Check for a login prompt and login locally to the Management Console or browse to the Field Effect Portal's Status page.

To note, issues with the power cable and green cable could cause it to go offline. The power cable, for obvious reasons, and the green cable as it provides internet connectivity. The yellow and blue cables pass network traffic to the sensor and should not cause issues with connectivity.

3.3: Reboot the device 

After checking on the YubiKey and cables, in case any of them created the issue, the sensor still might still need a reboot to come back online. For example, if the green cable was loose, the sensor may need a reboot to acquire an IP again. 

A reboot should normally consist of turning off the sensor off for 30 seconds and then powering it back on. The power buttons are on the back for the B1, E1 and E100 – for the Oskar, you will need to unplug the power cable to reboot it.

4: Check firewall rules and logs

If you can ping the sensor's IP address locally, the sensor is online - but it is unable to communicate back with Field Effect MDR. If this is the case, ensure that our relay connection is allowed as an outbound connection within your firewall(s) rules. 

You can use the Status Page to find your sensor's relay server hostname, that needs to be allowed as an outbound connection.

Firewall rules must be set to ensure that the sensor, and endpoint agents, can function correctly. The OpenVPN rule, if misconfigured, could cause connectivity issues.  

Example Scenario:

A firewall is blocking both UDP port 1194 (OpenVPN) and TCP port 443 (HTTPS) connectivity from your network to Field Effect datacenter.

This firewall rule should have already been set if the sensor was working previously, but if any changes were made to the firewall, it’s possible the rule was altered or removed.

Please refer to your firewall logs, as it would reveal if anything was being blocked – such as the required OpenVPN firewall rule. 

5: Check sensor Logs

To check on the logs, you will need to login to the Management Console at the sensor physically (with peripherals) or by an SSH connection. 

Feel free to do this independently, or by scheduling a call with the Field Effect support team. Please reach out to support@fieldeffect.com to get the credentials for your sensor, or to schedule a call to go through the logs.

6: Replace the sensor

If all the remediation steps above were taken and no solution was found, contact Support to discuss replacing the unit.