OWASP FSTM, stage 5: Analyzing filesystem contents

The operation of IoT and embedded devices relies on an operating system that stores configuration and executable files in a file system embedded in the firmware. This article focuses on analyzing filesystem contents, which can provide crucial information about device operation and vulnerabilities.

In the following sections, the general process of analyzing the contents of a file system is presented in different sections, starting with an understanding of the start-up process and the services that are executed, and then providing an overview of the most relevant file types on which to focus the search for information.

The fifth stage of the OWASP Firmware Security Testing Methodology aims to obtain as much information about the operation of the system as possible without executing any files, a further step in performing an IoT security audit. As a result of the process, the aim is to obtain a list of configuration files, keys, and executables of interest for later stages, as well as information about the boot process, the services that are executed, default users and other elements of interest for vulnerability exploitation.

In the previous stage, one or more file systems found in the device firmware have been extracted. These file systems can have different purposes and the directory tree of a Unix-like operating system will not always be found in them. However, it is quite common to find this type of file systems, and, in these cases, it is essential to study the elements they contain.

First, the directory tree contains the configuration files and executables that conduct the system boot process at the user level once the kernel boot has finished. Studying this process can provide information about the peripherals that the device expects, the services that run at startup, the users that are enabled and the initial configuration of the device.

After understanding the boot process, sufficient clues will be obtained to continue the analysis and arrive at executables and files of interest. After exhausting this analysis path, an automated file search is run for further study. In this process, configuration files, private keys, passwords embedded in the firmware and vulnerable and non-standard executables are searched for.

Finally, there are tools that automate part of the work, but whose results should always be checked and investigated manually, as they may produce false positives or miss important items.

The following sections detail analyzing filesystem contents steps and provide tips and examples for performing them.

Boot process

The starting point of the analysis is the study of the boot process, which refers to the tasks that are executed when entering the userspace, after the kernel boot.

Although there are different types of boot processes in Unix systems the main ones are:

• BSD: classic boot process of BSD operating systems, such as FreeBSD and OpenBSD, which executes the scripts /etc/rc, /etc/rc.local, /etc/rc.conf and the scripts contained in /etc/rc.d, although it depends on the system version.
• System V: a very popular boot process, implemented in BusyBox, among other systems, which reads the contents of the /etc/inittab configuration file and executes the scripts found in the /etc/init.d directory, depending on the configuration.
• systemd: typical boot process of modern Linux systems, although not common on embedded devices, which configures the boot by launching services according to the configuration files in the directory /etc/systemd/

To continue with the analysis of the boot process, it is therefore necessary to identify the type of boot process of the device. This information can be displayed during boot by the terminal, on systems that have one at runtime, but it can also be found by analyzing the init executable, which is the first to be executed when the userspace starts. This is usually found in /sbin/init but can be modified via kernel’s init option. If the init= string is not found in the firmware image, the executable is located in /sbin/init.

With BusyBox and System V, a common case for IoT devices, /sbin/init is a symbolic link to the /bin/busybox executable, which performs the boot operations:

$ ls -l sbin/init
lrwxrwxrwx 1 root root 14 ago 2 21:42 sbin/init -> ../bin/busybox

With systemd, the file is usually a symbolic link to /lib/systemd/systemd:

$ ls -l sbin/init
lrwxrwxrwx 1 root root 22 jul 13 16:24 sbin/init -> ../lib/systemd/systemd

On other systems, /sbin/init is an ELF executable:

$ file sbin/init
sbin/init: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, no section header

It can be analyzed with common executable analysis techniques: file, strings, and reverse engineering tools such as Ghidra or IDA.

Once the boot system has been identified, the corresponding configuration files and startup scripts are analyzed. In this example firmware, extracted from an IoT device, /sbin/init is a symbolic link to the BusyBox executable and it refers to the /etc_ro/inittab and /etc/inittab files, confirming that the boot process is System V:

$ strings bin/busybox | grep inittab
Bad inittab entry at line %d
/etc_ro/inittab
reloading /etc/inittab

Analyzing the contents of /etc_ro/inittab, it is found that it runs the /etc_ro/rcS script and starts /bin/sh for the ttyS1 terminal:

$ cat etc_ro/inittab
::sysinit:/etc_ro/rcS
ttyS1::respawn:/bin/sh

/etc_ro/rcS has the following content:

#!/bin/sh
#kent_chang 20091208, Enable USB power by turn GPIO_11 on and keep it on
##gpio l 11 4000 0 1 0 4000
mount -a
#cat /etc_ro/motd
mkdir -p /var/run
mkdir -p /var/log
mkdir -p /var/dev
mkdir -p /var/run/ppp
mkdir -p /proc/bus/usb
mkdir -p /proc/sys/kernel
mkdir -p /etc/mtab
mkdir -p /etc/modprobe.d
mkdir -p /etc/ppp/peers
mkdir -p /etc/Wireless/RT2870STA
mkdir -p /var/run/ppp
mkdir -p /var/spool
mkdir -p /var/spool/cron
mkdir -p /var/spool/cron/crontabs
makedevlinks.sh
nvram_daemon&
mkdir -p /var/log
ln -sf /etc_ro/fstab /etc/fstab
#ln -sf /etc_ro/udev /etc/udev
ln -sf /etc_ro/init.d /etc/init.d
ln -sf /etc_ro/Wireless/RT2870STA/RT2870STA.dat /etc/Wireless/RT2870STA/RT2870STA.dat

wan_dual_wan_backup=`nvram_get 2860 wan_dual_wan_backup`
wan_ip_assignment=`nvram_get 2860 wan_ip_assignment`
if [ $wan_dual_wan_backup -eq 99 -a $wan_ip_assignment -eq 3 ]; then
ln -sf /etc_ro/udev /etc/udev
else
mkdir -p /etc/udev
mkdir -p /etc/udev/rules.d
cp /etc_ro/udev/rules.d/70-* /etc/udev/rules.d/
cp /etc_ro/udev/udev.conf /etc/udev/
fi

#wan_ip_assignment=`nvram_get 2860 wan_ip_assignment`
modprobe lm
modprobe dwc_otg
# modprobe usb-storage
/bin/sh -x /etc/init.d/udev.sh > /dev/console
#nvram_daemon&
goahead >/dev/console&
#for telnet debugging
telnetd &

Among the points of interest of the script, the following is noted:

• Runs the makedevlinks.sh script.
• Starts a nvram service.
• Creates symbolic links of /etc_ro/fstab and /etc_ro/init.d in /etc
• Call the script /etc/init.d/udev.sh and send the result to /dev/console
• Start goahead and send the result to /dev/console
• Start a telnetd service.

The script /etc/init.d/udev.sh initializes the /etc/group and /etc/passwd files, and that the root user can log in without a password:

$ cat etc_ro/init.d/udev.sh
#!/bin/sh
#Tom add for 3G device manager 05-11-2009 begin
wan_3g_model=`nvram_get 2860 wan_3g_model`

…
echo “root::0:0:root:/root:/bin/ash” >> /etc/passwd
echo “bin:x:1:1:bin:/bin:/sbin/nologin” >> /etc/passwd
echo “lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin” >> /etc/passwd
echo “uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin” >> /etc/passwd
echo “ntp:x:38:38::/etc/ntp:/sbin/nologin” >> /etc/passwd
echo “ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin” >> /etc/passwd
echo “nobody:x:99:99:Nobody:/:/sbin/nologin” >> /etc/passwd
echo “Starting udev…” > /dev/console
#/sbin/udevd –daemon –debug-trace
/sbin/udevd –daemon
#/sbin/udevsettle
# get date/time is less then 2010/01/01
nowdate=”`date -u +\”%Y%m%d\”`”
[ “$nowdate” -lt “20110101” ] && date -u 010100002011.00 && echo “set date/time -> `date`”

wan_dual_wan_backup=`nvram_get 2860 wan_dual_wan_backup`
wan_ip_assignment=`nvram_get 2860 wan_ip_assignment`
[ $wan_dual_wan_backup -eq 99 -a $wan_ip_assignment -eq 3 ] && /sbin/3gd &
[ $wan_dual_wan_backup -ne 99 ] && killall -q 3gd
[ “`nvram_get 2860 wanConnectionMode`” -eq “USB-WLAN” ] && /sbin/3gd &
systime=`bc_nvram_get 2860 acc3g_systime`
date -s $systime
sync
date -u > /dev/console
date -u +”%m%d%H%M%Y.%S” >/etc/ntp.date
synctime=`cat /etc/ntp.date`
sync
#Tom add for 3G device manager 05-11-2009 end

In addition to the study of the services raised at startup, the files in the /etc_ro/init.d directory are the configuration files of the services available on the system:

$ ls -l etc_ro/init.d
total 248
-rwxr-xr-x 1 dummy dummy 191056 jul 20 2017 3g.sh
-rwxr-xr-x 1 dummy dummy 3650 jul 20 2017 cron.sh
-rwxr-xr-x 1 dummy dummy 2317 jul 20 2017 hostname.sh
-rwxr-xr-x 1 dummy dummy 5813 jul 20 2017 load_balance.in
-rwxr-xr-x 1 dummy dummy 189 jul 20 2017 load_balance_re.sh
-rwxr-xr-x 1 dummy dummy 6884 jul 20 2017 load_balance.sh
-rwxr-xr-x 1 dummy dummy 3337 jul 20 2017 ntp.sh
-rwxr-xr-x 1 dummy dummy 1934 jul 20 2017 static_ip.sh
-rwxr-xr-x 1 dummy dummy 2752 jul 20 2017 udev.sh
-rwxr-xr-x 1 dummy dummy 244 jul 20 2017 update_flash_systemtime.sh
-rwxr-xr-x 1 dummy dummy 3507 jul 20 2017 wan_l2tpc.sh
-rwxr-xr-x 1 dummy dummy 3136 jul 20 2017 wan_pppoe.sh
-rwxr-xr-x 1 dummy dummy 3329 jul 20 2017 wan_pptpc.sh
-rwxr-xr-x 1 dummy dummy 2541 jul 20 2017 wan_udhcpc.sh

A detailed study of the system services is beyond the scope of this article, but it can be especially useful. For example, in the file /etc_ro/init.d/cron.sh there is a call to the /sbin/custom_config.sh script:

$ cat etc_ro/init.d/cron.sh
#!/bin/sh

. /sbin/custom_config.sh
…

In which there are default configuration parameters, including the default password for the web administration console:

$ cat sbin/custom_config.sh
CINFO_WEB_Admin_Password=”admin”
CINFO_Manufacturer_Title=”ALFA Networks”
CINFO_Manufacturer_Name=”ALFA Networks”
CINFO_Manufacturer_URL=”https://www.alfa.com.tw”
CINFO_Model_Title=”ALFA-R36″
CINFO_Model_Name=”ALFA”
CINFO_Model_Number=”R36″
CINFO_Model_Description=”USB Wi-Fi Router”
CINFO_Model_URL=”https://www.alfa.com.tw”
CINFO_AUTO_REBOOTING_TIME=”0″
CINFO_Country_Region=”5″
CINFO_Country_RegionABand=”7″
CINFO_Country_Code=””
CINFO_Host_Name=”R36″
CINFO_Domain_Name=”alfa.com.tw”
CINFO_NTP_Server_IP=”time.nist.gov”
CINFO_NTP_Sync=”6″
CINFO_AP_DEFAULT_CHANNEL=”1″
CINFO_AP_WMM_Capble=”0″

In this case, the information obtained from the analysis of the boot process is sufficient to search for an open telnet service or serial port to open a terminal, where you can log in as root without a password, or to try to access the web administration console with the default password. In addition, executables that participate in service initialization, such as goahead, have been identified and might be of interest for further analysis.

It’s common for the study of the boot process to lead to a much deeper understanding of the operation of the device and can uncover important vulnerabilities, so it is useful to obtain as much information as possible about the initial configuration and services on the system.

Searching files of interest

After analyzing the startup process and with a clearer idea of the available services and their general operation, it can be especially useful to perform an exhaustive search in the file system to find configuration files, executable keys, scripts, and other types of files that do not depend directly on the startup process.

Some items of interest to look for during this phase are:

• Outdated, insecure or misconfigured servers.
• Default or insecurely stored credentials, such as usernames, passwords, API keys or SSH keys.
• Default API services and details of their operation.
• Updating functionality and services.
• Source code and startup scripts.
• Compiled executables.

From these elements, vulnerabilities can be found in the system that allow the execution of different types of attacks, such as system intrusion, code execution or denial of service.

During the analysis, the contents of the directories should be checked in an orderly fashion, starting with the locations of interest in the file system, such as:

• /etc/passwd: list of the system users and details about their configuration
• /etc/shadow: contains user passwords, stored in the form of a hash digest along with a cryptographic salt, and can provide information about users with permission to log in.
• /etc/group: list of system groups
• /etc/fstab: list of volumes that are mounted at system startup and their configuration
• /etc/init.d: start-up scripts, in case they have not been studied in the previous stage
• /etc/runlevels and/etc/inittab: configuration of the start-up scripts, in case they have not been studied in the previous phase
• /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin and /usr/local/sbin: executables and tools
• /etc: other configuration files
• /var/log: system and application logs
• /root and/home: root and user home directories

After an initial search in the usual locations, it can be especially useful to run automatic searches with tools such as find, which allows you to search for files based on patterns in the entire file system or in a directory.

The main criteria for searching for files of interest are as follows:

• Files related to SSL and encryption keys, with .pem or .crt extension.
• Configuration files, with a .conf extension.
• Scripts, with .sh extension.
• Files with execution permissions.
• Binary files, with no format indication, with .bin extension.
• Known web servers, such as nginx or apache.
• Executables and services that expose network interfaces such as ssh, telnet, tftp or dropbear.

In a more exhaustive search, the contents of the configuration files can be inspected for interesting data such as:

• Keywords like admin, pass, password, remote or key.
• Vulnerable and unsafe C functions, such as gets, strcpy, stpcpy, strcat, printf or sprintf.
• URLs, IP addresses or email addresses.

In the example firmware, when searching for files with .sh extension, different interesting files are found:

$ find . -name ‘*.sh’
./etc_ro/init.d/static_ip.sh
…
./sbin/config.sh
./sbin/chpasswd.sh
./sbin/hotspot.sh
./sbin/pppoe.sh
./sbin/vpn-passthru.sh
./sbin/automount.sh
…

Among them, /sbin/config.sh contains the default configuration of the device, where the serial terminal configuration detected during the boot analysis can be found:

$ cat sbin/config.sh
CONFIG_MIPS=y
…
CONFIG_CMDLINE=”console=ttyS1,115200n8 root=/dev/mtdblock4″
…

The configuration line shown indicates that the ttyS1 serial terminal receives connections with a baud rate of 115,200 baud.

Automation tools

For this tedious task of search and identification, there are public tools available that perform an automated analysis of the contents of a firmware file system, such as Firmwalker, FACT or EMBA.

Firmwalker

Firmwalker is a free software tool developed in bash script to automate the search for items of interest in a file system of a firmware extracted or mounted in the analysis environment. It can be found in the project’s repository on GitHub: https://github.com/craigz28/firmwalker.

Its basic usage is as simple as executing the following command:

$ ./firmwalker filesystem_root

Where filesystem_root is the root of the file system. The result is like the following:

$ ./firmwalker.sh ~/firmware/squashfs-root
***Search for password files***
##################################### passwd
t/usr/bin/passwd

##################################### shadow

##################################### *.psk
…

***Search for files***
##################################### *.conf
t/etc_ro/udev/udev.conf
t/etc_ro/lld2d.conf
t/etc_ro/radvd.conf
t/etc_ro/miniupnpd.conf
…
——————– pwd ——————–
t/bin/wpa_supplicant
t/bin/openssl
t/bin/busybox
t/lib/udev/path_id
t/lib/libuClibc-0.9.28.so
t/usr/sbin/msmtp
…

——————– telnet ——————–
t/etc_ro/l7-protocols/telnet.pat
t/etc_ro/rcS
t/bin/busybox
t/lib/libcrypto.so.0.9.8
…

The result produced by the tool, as in all cases of automatic tools, should be considered as indications of points of interest in the firmware, which will have to be further analyzed to search for vulnerabilities. It serves as a good starting point for an analysis of the elements shown, not as a final report.

The Firmware Analysis and Comparison Tool

The Firmware Analysis and Comparison Tool or FACT is another analysis tool that aims to automate most of the complete firmware analysis process, including content identification, file system extraction and analysis. The project can be found in the Github repository: https://github.com/fkie-cad/FACT_core.

FACT has functions for key and certificate detection, detection of weak implementations listed in the Common Weakness Enumeration (CWE) and comparison of known file versions, among others.

The framework is available in the FACT_docker docker image or can be installed via a script. It’s started by running start_all_installed_fact_components as root and accessed via a web located on ports 5000 and 80: https://localhost:5000 and https://localhost .

$ sudo ./start_all_installed_fact_components

Embedded Analyzer

Embedded Analyzer or EMBA is another framework, this one developed in bash scripting, intended to automate the entire firmware analysis process. The project is available in the GitHub repository: https://github.com/e-m-b-a/emba.

The EMBA project gathers multiple previously available tools and has multiple functionalities, such as file system extraction, static analysis of executables or identification of known vulnerabilities.

For installation, EMBA recommends the use of a Kali Linux or Ubuntu virtual machine and provides an installation script, downloading a docker image on which the framework runs. The execution is performed with the following command:

$ sudo ./emba.sh -l ./log_file -f /firmware_image

Also available is EMBArk, an adaptation of EMBA with a web interface oriented to a business environment.

In IoT devices that have an operating system and contain a file system in the firmware, which is usually the case, analysis of the content of the file system is essential to understand the operation and potential vulnerabilities.

Although automated tools are available to perform the initial recognition of points of interest, manual analysis and the use of tools such as find, file, strings, and hexadecimal editors is still necessary to find vulnerabilities where automation can’t reach.

Detailed analysis of the device’s boot process yields a wealth of information about its operation, the services running and even possible access points.

Nevertheless, the analyst’s experience and knowledge are critical to identify and follow clues that may reveal insecure configurations, vulnerable services or executables and other security flaws.

References

• https://github.com/scriptingxss/owasp-fstm
• https://github.com/craigz28/firmwalker
• https://github.com/fkie-cad/FACT_core
• https://github.com/e-m-b-a/emba