ACS Kryten How-to Guide

A guide to running Flightkit software on the AAC Clyde Space Kryten onboard computer.

1. Introduction

The AAC Clyde Space (ACS) Kryten onboard computer (OBC) is built around an ARM Cortex-M3-based system-on-chip (SoC) with integrated flash, I2C, SPI, UART, CAN and GPIO peripherals. The board hosts additional persistent memory and telemetry peripherals.

Flightkit supports all of the key parts of the OBC.

This guide provides an overview of:

  • How images are stored on the Kryten.

  • How images are run on the Kryten.

  • The main component types included in the FlightkitFoundation bundle which enable you to interact with the board’s drivers.

  • How to build missions for the Kryten.

  • How to program the Kryten.

  • How to run unit tests on the Kryten.

  • How to troubleshoot the Kryten.

2. Technical Background

In this section we give an overview of the Kryten support provided with FlightkitFoundation. This includes details on how the Kryten’s onboard memory is used, how different software images are managed, and the different component types we provide for interacting with the board’s peripherals.

2.1. The Memory Map

The memory map of the Kryten is an extension of the standard map created by the SmartFusion2 device that the OBC is based on (which is in turn a specialisation of the standard ARM Cortex-M3 memory map).

The main code execution area is at the bottom of the memory map. This region is actually a mirror of either the internal flash, or the MRAM code space, depending on whether the Kryten is running a failsafe or primary image.

Older Kryten computers only had 256KiB of onboard flash - Flightkit supports this via the KrytenFailsafe256 build configuration. Check with ACS to find out whether you have a Kryten with 256KiB or 512KiB of flash.
Range Size Usage MPU Attributes

0x00000000
0x0007FFFF (failsafe) 0x000FFFFF (primary)

512 KiB (failsafe)
1 MiB (primary1)

Execution mirror. Code is executed from this section. At boot the relevant code section (either failsafe, primary1 or primary2) is "mirrored" here by the board support package.

RO

The internal 64KiB SRAM is used for critical code and data which must survive an MRAM error (and therefore an MRAM power cycle). Additionally, the system interrupt stack is also placed in SRAM.

Range Size Usage MPU Attributes

0x20000000
0x20003FFF

16 KiB

SRAM code. Code which must be executable when the MRAM is not available is loaded here at boot. e.g. code to control formatting of the MRAM devices

RO

0x20004000
0x20007FFF

16 KiB

SRAM data, including from CRT. Used for special data which must be accessible when the MRAM is not available.

RW, XN

0x20008000
0x2000FFFF

32 KiB

Main stack (used only for interrupt handling)

RW, XN

The internal non-volatile flash memory is used to hold the failsafe image.

Range Size Usage MPU Attributes

0x60000000
0x6007FFFF

512 KiB

On board flash. Used for storing the failsafe software image. Execution is from the mirrored section starting at 0x00000000.

RW, XN

The non-volatile MRAM is used for four distinct purposes:

  1. Primary image storage

  2. Main working memory (overwritten on each boot)

  3. Record stores for critical data and event logging

  4. Persistent component configuration data

Range Size Usage MPU Attributes

0xA0000000
0xA00FFFFF

1 MiB

Primary1 software image storage. Execution is from the mirrored section starting at 0x00000000.

RO, XN

0xA0100000
0xA01FFFFF

1 MiB

Primary2 software image storage. Execution is from the mirrored section starting at 0x00000000.

RO, XN

0xA0200000
0xA03FFFFF

2 MiB

Working MRAM. This area is used for .data and .bss sections, whether running failsafe, primary1 or primary2.

RW, XN

0xA0400000
0xA077FFFF

3.5 MiB

Persistent MRAM used for data logging

RW, XN

0xA0780000
0xA07FFFFF

512 KiB

Persistent MRAM used for configuration data

RW, XN

Additionally, the Kryten provides 4 GiB of flash but this is not directly memory mapped. This is typically used to host a file system.

2.2. The Boot Process

As indicated in the previous section, the Kryten hardware has two distinct memories which can be used to hold flight software. A single failsafe image can be stored in the SoC’s embedded flash memory, and two primary images can be stored in its EDAC-protected MRAM.

The flash memory is more limited than MRAM, so in practice, failsafe images tend to be cut down versions of the primary images.

Failsafe images should not be used for nominal operations during a mission, they should only be used as a means to recover the satellite if the primary images become corrupted or unstable. They should therefore be kept simple to ensure correct operation.

Primary images on the other hand are upgradeable, so they can safely include more complex functionality. These images are typically responsible for carrying out the mission.

The following procedure is used to select and boot flight software images:

  • From hard reset:

    1. Execute the bootloader code in the failsafe image.

  • Select the next boot image:

    1. Iterate through each of the images' boot configuration records.

    2. Select the image which is stable and has the highest priority as the next boot image.

    3. Verify the next boot image’s data against its stored CRC. The failsafe image is immutable and always passes this check.

    4. If the CRC mismatches, mark the image as unstable and select another next boot image.

  • Prepare image for boot

    1. If the next boot image is not the failsafe image, mark it as unstable. The booted image may mark itself as stable again after booting. The stability flag mechanism prevents boot loops of unstable primary images.

  • Boot the image

    1. If the next image is not the failsafe image, boot to it. Otherwise continue execution from the failsafe image.

The drivers.kryten.periph.OBC component type can be used to control the contents of the primary images, and to manipulate the boot configuration records to control which image will be booted next. This will be discussed further in the following sections.

2.3. Component Types

The FlightkitFoundation bundle includes a number of Kryten-specific component types which can be used by your flight software to interact with the peripherals and drivers on board the computer.

2.3.1. The OBC Component Type

The drivers.kryten.periph.OBC component type provides access to various core functions of the Kryten hardware. This includes powering on and off onboard subsystems, querying firmware and revision information, and controlling the running of flight software images.

Given the latter point, it is vital that you include an instance of this component type in your failsafe image slot if you wish to be able to switch images during operation.

The component type uses the following convention when referring to the different image slots:

  • 0: The failsafe image

  • 1: The primary1 image

  • 2: The primary2 image

To prevent boot loops from occurring, primary images are marked as unstable during the boot process. This prevents them being considered during the next boot cycle. Automation can be set up in the deployment type to mark them as stable again following a successful boot. Alternatively, the operator can carry this out manually when contact with the spacecraft is achieved.

2.3.2. The Watchdog Component Type

The Watchdog component manages the Kryten’s onboard 555 timer-based watchdog circuit.

It has two periodic tasks, a restore task and a kick task.

The restore task resets the number of times the watchdog timer must be kicked before a watchdog reset is performed. This must be set to the lowest task priority in the deployment type.

The kick task handles the kicking of the watchdog timer. This must be set to the highest task priority in the deployment type.

This combination of task priorities ensures that the watchdog timer continues to be kicked so long as there is enough "credit" available. If you wish to learn more, refer to the component type’s documentation.

2.3.3. Pin Configuration, GPIO and Peripherals

The Kryten’s IO pins on the CSK can be configured for many different functions. At start up, the flight software configures all pins on the CSK to be inputs. During the deployment initialisation phase, pin functions may be altered depending on which driver component instances exist within the deployment instance, and how they have been initialised. Some instances may require the use of specific pins to fulfil their obligations:

  • drivers.kryten.io.bus.Serial and drivers.kryten.io.bus.FastSerial may claim pins for serial TX and RX.

  • drivers.kryten.io.bus.SPIController and drivers.kryten.io.bus.SPIPeripheral may claim pins for MISO, MOSI, SCLK and SS.

  • drivers.kryten.io.GPIO may claim pins for general I/O purposes.

  • drivers.kryten.periph.PPS may claim pins for use as PPS input and output.

Note that pins are claimed by component instances unconditionally. If component instance B is instantiated after component instance A, then B’s local initialisation will happen after A’s. This means that any pins that B claims which A has already claimed will end up having B’s settings applied. Debug output during deployment initialisation can be used to check for collisions in pin allocation.

The Kryten board provided in the FlightkitFoundation bundle references pins by their position on the CSK as opposed to the inconsistent numbering and naming provided by ACS. For example H1.1 is referred to using the pin identifier BOARD_PINMUX_PINID_H1_1 as opposed to GPIO_0. Similarly H1.17 is referred to as BOARD_PINMUX_PINID_H1_17 as opposed to GPIO_10.

The available pin configuration options are determined by the version of firmware that is running on the Kryten.

The Kryten board provided in the FlightkitFoundation bundle is configured such that it targets Kryten firmware 7725 RevG m2p0 by default. This setting can be overwritten for your own missions by redefining the BOARD_CONFIG_FIRMWARE_ID symbol in your mission’s lib_config/meson.build file to your desired version number. Note that you must set this variable using hex directly. To target 7725 RevI m1p0 firmware, add the following line to your lib_config/meson.build file:

boards_Kryten_c_args += '-DBOARD_CONFIG_FIRMWARE_ID=0x1E2D0910'

The first 4 characters (1E2D) are the part number (7725), the next 2 (09) are the major revision (RevI) and the final two (1 and 0) are the minor and patch revisions (1 and 0).

3. Working With the Kryten

This section outlines various procedures which you will need to use when working with the Kryten.

3.1. Installing the Toolchain

Flightkit support for the ACS Kryten requires you to install version 10.3-2021.10 of ARM’s arm-none-eabi toolchain.

This is the same version as required to use Flightkit’s EnduroSat OBC support, so if you have already installed it via the EnduroSat OBC How-to Guide then you can skip this section.

A very similar version of this toolchain is available directly from the Ubuntu package repositories. However, it supplies a C standard library which is built without support for features which our Kryten board support code uses.

As a result you should not use the toolchain in the Ubuntu package repositories.

The arm-none-eabi toolchain can be found on this page, or downloaded directly by clicking here.

Once downloaded:

  1. Extract the archive. The /opt directory is a common location used for storing external toolchains.

  2. Add the extracted …​/gcc-arm-none-eabi-10.3-2021.10/bin directory containing the toolchain executables to your PATH.

  3. Verify that the toolchain was successfully added to the path by restarting your shell and running the following command:

    arm-none-eabi-gcc --version

3.2. Building the Demo Mission

With the tooling successfully installed, you can build flight software to run on the Kryten. The following steps will guide you through the process for building the KrytenDemo mission from the FlightkitFoundation bundle:

  1. Create a new workspace for building the mission in.

  2. Require the FlightkitFoundation bundle so that the tooling can access its contents by running the following command:

    hfk workspace bundle require FlightkitFoundation

  3. Build the KrytenDemo mission from the FlightkitFoundation bundle by running the following command:

    hfk mission build KrytenDemo

This builds all the deployment instances in the mission, and generates a mission database for interacting with them via Helix Lab. The binaries and database can be found in output/bundles/FlightkitFoundation/X.Y.Z/missions/KrytenDemo/, where X.Y.Z is the version of FlightkitFoundation you are using.

The KrytenDemo mission is configured to build deployment instances for each of the Kryten’s image slots.

When developing missions of your own, you can configure them to target whichever image slots you desire by setting your deployment instances' <BuildConfig> elements accordingly.

The list of available build configurations provided by a particular bundle can be determined by inspecting the bundle’s documentation.

3.3. Connecting to the Kryten

There are three main connections that you will need to set up when working with the Kryten: the telemetry and telecommand (TMTC) link, the debug link and the programmer.

The following sections will explain how to set up each one.

3.3.1. TMTC Link

The TMTC link is used to transport telemetry and telecommand (TMTC) packets between the Kryten and the PC running Helix Lab.

The deployment instances within the KrytenDemo mission are configured to provide the TMTC link using UART A with LVTLL signal levels.

To set up the TMTC link for the KrytenDemo deployment instances:

  1. Connect the TX line of a suitable serial to USB adapter to pin 33 of the H1 header.

  2. Connect the RX line of the same serial to USB adapter to pin 35 of the H1 header.

  3. Connect the ground line of the same serial to USB adapter to one of the ground pins on the H1 header.

  4. Connect the USB side of the serial to USB adapter to the host PC running Helix Lab.

The deployment instances in the KrytenDemo mission are configured with the following TMTC serial settings:

  • 57600 baud

  • 8 data bits

  • No parity bit

  • 1 stop bit

  • No flow control

Helix Lab is only capable of exchanging TMTC packets with a spacecraft via a TCP server. For this reason, you will also need to configure your host PC to forward packets destined for the TCP port that Helix Lab transmits to to the serial port that your TMTC link is connected to, and vice versa. There are many ways this can be achieved, however we recommend you use a tool such as ser2net or socat. If you are unfamiliar with how to use these tools, refer to their documentation online.

3.3.2. Debug Link

The debug link is used for transporting debug information from the Kryten to the PC where it can be displayed.

The deployment instances within the KrytenDemo mission are configured to use UARTD with LVTTL voltage levels for debug output.

To set up the debug link for the KrytenDemo deployment instances:

  1. Connect the RX line of a suitable serial to USB adapter to pin 8 of the J6 header.

  2. Connect the ground line of the same serial to USB adapter to one of the ground pins on the J6 header.

  3. Connect the USB side of the serial to USB adapter to the host PC.

To view the debug produced by the board on the host PC run a terminal emulator, such as minicom.

The deployment instances in the KrytenDemo mission are configured with the following debug serial settings:

  • 57600 baud

  • 8 data bits

  • No parity bit

  • 1 stop bit

  • No flow control

You must apply these settings to your terminal emulator so that it interprets the incoming data correctly. The debug produced by the board uses linux line endings, so you may also need to enable implicit carriage returns in your terminal emulator to view the data properly.

3.3.3. Programmer

The programmer is used to program binaries to the board.

Either an Olimex ARM-USB-TINY-H JTAG or a SEGGER JLink programmer can be used to program the ACS Kryten.

Both programmers come with their own set of hardware which consists of a USB-A to USB-B cable, the programmer itself, and a female to female JTAG ribbon cable.

To set up the programmer link:

  1. Connect the USB-A side of the USB-A to USB-B cable to the host PC that the binaries are being programmed from.

  2. Connect the USB-B side of the USB-A to USB-B cable to the programmer.

  3. Connect one side of the female to female JTAG ribbon cable into the programmer.

  4. Connect each of the required pins from the other side of the female to female JTAG ribbon cable to their respective pins on the Kryten’s J5 header.

3.4. Programming the Kryten

Failsafe and primary images must be programmed via the Olimex ARM-USB-TINY-H JTAG or SEGGER JLink programmer. Flightkit includes a programming script for the Kryten which provides a simple front-end to the OpenOCD tool. OpenOCD is installed when you install Flightkit.

Primary images cannot yet be uplinked using Helix Lab via the TMTC link. Support for uplinking primary images with Helix Lab will be added in future.

The following sections explain how to program failsafe and primary images. The instructions provided assume that you are programming the board from the same machine that you have used to build the binaries.

3.4.1. Programming a Failsafe Image

There must always be a failsafe image programmed on the Kryten platform. To boot into a primary image, a failsafe image is required both as the starting point, and a fall-back.

The following procedure demonstrates how to program a failsafe image using the KrytenDemo mission’s OBCDemo deployment instance. The same approach can be used for programming any binary that has been built for the failsafe image slot.

To program a binary to the failsafe image slot:

  1. Make sure your Kryten is powered from a stable power supply. If you are powering the Kryten via an ACS EPS make sure the EPS watchdog timer is increased to avoid power interruptions during programming.

  2. Program the binary using the following command, ensuring to replace X.Y.Z with the version of FlightkitFoundation you are using. If using the SEGGER JLink programmer, include --debugger=jlink within the command:

    hfk-kryten-program failsafe output/bundles/FlightkitFoundation/X.Y.Z/missions/KrytenDemo/OBCDemo/Default/Default.bin

  3. To verify that the image has been programmed successfully:

    • Open a terminal emulator, such as minicom, on the host PC to display the board’s debug output.

    • Power cycle the board to force a reboot.

    • Inspect the debug output displayed on the terminal emulator. If programmed successfully, it should show similar output to what is shown below.

      INF: Deployment.c:599 Running deployment:
INF: Deployment.c:600 - Deployment instance: Default
INF: Deployment.c:601 - Deployment type:     acs.kryten.OBCDemo
INF: Deployment.c:602 - Target:              OBCDemo
INF: Deployment.c:603 - Mission:             KrytenDemo
INF: Deployment.c:773 Deployment initialisation successful

3.4.2. Programming a Primary Image

Primary images can be programmed using a similar approach to programming failsafe images, with the addition of one extra step.

The following procedure demonstrates this by programming the Primary1 deployment from the KrytenDemo mission’s OBCDemo target into the Kryten’s primary1 slot. This mission is found in the FlightkitFoundation bundle.

Note that binaries programmed to the primary1 slot must be built for that slot. We provide the KrytenPrimary1 build configuration for this purpose. To program a binary to the primary2 slot, the KrytenPrimary2 build configuration must be used instead. Build configurations are set on a per-deployment basis in the mission.xml.

To program a binary to the primary1 image slot:

  1. Generate a tag file for the binary by running the following command. The purpose of the tag file is to provide additional metadata such as binary length and CRC, which is required when programming to the primary image slots:

    hfk-kryten-gen-crc --realign --tag output/bundles/FlightkitFoundation/X.Y.Z/missions/KrytenDemo/OBCDemo/Primary1/Primary1.bin

  2. Program the binary using the following command, ensuring to replace X.Y.Z with the version of FlightkitFoundation you are using. If using the SEGGER JLink programmer, include --debugger=jlink within the command. Also ensure the slot which you are programming to corresponds with the build config that was used to build the binary:

    hfk-kryten-program primary1 output/bundles/FlightkitFoundation/X.Y.Z/missions/KrytenDemo/OBCDemo/Primary1/Primary1.bin

3.5. Switching Images

The following section details the procedure for switching between different images during operation. It explains how you can switch from a failsafe image to a primary1 image, however the same process can be applied to switch between any two images.

Once a valid primary image has been successfully programmed/uplinked to the primary1 slot, it can then be booted into by completing the following steps:

  1. Ensure the Kryten board is running the failsafe image, and connect to it using Lab.

  2. (Optional) Find the drivers.kryten.periph.OBC instance’s imageValid parameter. Set both the first row and last row fields to 1, then perform a get on the parameter. This will return a flag indicating if the stored CRC for the image in the primary1 slot matches its expected value. In other words, it will confirm whether the image is valid. If it is found to be invalid, any attempt to boot into it will fail.

  3. Find the drivers.kryten.periph.OBC instance’s imageIsStable parameter. Set both the first row and last row fields to 1, then perform a set on the parameter, passing the value 1. This marks the image in the primary1 slot as stable. Images must be marked as stable for them to be considered during the boot process. The failsafe image is permanently marked as stable, so it will always be booted into if none of the other images are marked as stable.

  4. Find the drivers.kryten.periph.OBC instance’s imagePriority parameter. Set both the first row and last row fields to 1, then perform a set on the parameter, passing the value 2. This sets the priority value of the image in the primary1 slot to 2. The image with the highest priority value will be booted into during the boot process, so long as it is also marked as stable. The priority value of the failsafe image is always set to 1, so if a primary image is intended to be booted into, its priority value must be set to a value greater than 1.

  5. (Optional) Find the drivers.kryten.periph.OBC instance’s nextBootImage parameter. Perform a get on the parameter. This will return the index of the image that the bootloader will attempt to boot into during the next boot cycle. It does not guarantee that the specified image will be booted into as it may fail if the image is not valid.

  6. Find the drivers.kryten.periph.OBC instance’s reset action and invoke it. This will trigger the boot process resulting in the nextBootImage being booted into. If it fails to boot into the nextBootImage for whatever reason, the boot process will fall back onto booting the next image in the list that has the highest priority, is found to be valid and is marked as stable. This process will repeat until an image is successfully booted into, ending with the failsafe image if required.

  7. (Optional) When booting into a primary image, the boot process will automatically clear the stability flag of the image it is booting into to prevent boot loops from occurring in the event that the primary image is corrupt or faulty. If you would like the same image to be booted into again during the next boot process, it can be marked as stable again by invoking the markCurrentImageStable action.

3.6. Running Unit Tests on the Kryten

When building unit tests in Flightkit, the tooling produces a binary which can be run on the platform in the same way that deployment instances can. The main difference is that unit tests are built on a 'per component type' basis, whereas deployment instances are built on a 'per mission basis'. This small detail requires unit tests to be built using a slightly different process to deployment instances, the steps of which are shown below.

The following steps demonstrate how unit tests for the Counter component type can be built and run on the Kryten platform. The same process can be followed for any component type:

  1. Build the component type’s unit tests by running the following command. Notice that this example builds them for running in the primary1 image slot. Unit tests can be built to run in any image slot, however the primary slots can be programmed more quickly since they are stored in MRAM as opposed to the failsafe slot which is stored in flash, so we recommend taking this approach:

    hfk component-type test Counter --build-config="KrytenPrimary1" --build-only

  2. Since the binary is going to be run in a primary image slot, a tag file must be generated for it. When generating the tag file, we can also mark the image as stable and set its priority to be greater than the failsafe image so that it is booted into instantly during start up. Run the following command to accomplish this, ensuring to replace X.Y.Z with the version of FlightkitFoundation you are using:

    hfk-kryten-gen-crc --realign --tag --priority=2 --stable output/bundles/FlightkitFoundation/X.Y.Z/library/component_types/Counter/KrytenPrimary1/CounterTest.bin

  3. Open a terminal emulator, such as minicom, to display the debug output from the board. This will be used to display the unit test results.

  4. Program the binary using the following command, ensuring to replace X.Y.Z with the version of FlightkitFoundation you are using. If using the SEGGER JLink programmer, include --debugger=jlink within the command:

    hfk-kryten-program primary1 output/bundles/FlightkitFoundation/X.Y.Z/library/component_types/Counter/KrytenPrimary1/CounterTest.bin

The unit tests should run automatically once programming is complete. Output similar to what is shown below should be displayed on your terminal emulator indicating the tests that have been run and their completion status.

library/component_types/Counter/test/Counter_Test.c:134:Counter_Test_initAndFini:PASS
DBG: Counter.c:215 CounterUnderTest: Reset action called
library/component_types/Counter/test/Counter_Test.c:141:Counter_Test_resetSuccessful:PASS
DBG: Counter.c:215 CounterUnderTest: Reset action called
library/component_types/Counter/test/Counter_Test.c:159:Counter_Test_incrementSuccessful:PASS
DBG: Counter.c:215 CounterUnderTest: Reset action called
library/component_types/Counter/test/Counter_Test.c:177:Counter_Test_decrementSuccessful:PASS
library/component_types/Counter/test/Counter_Test.c:203:Counter_Test_getConfigInsufficientResources:PASS
library/component_types/Counter/test/Counter_Test.c:218:Counter_Test_getConfigSuccess:PASS
library/component_types/Counter/test/Counter_Test.c:233:Counter_Test_setConfigInvalidParam:PASS
library/component_types/Counter/test/Counter_Test.c:249:Counter_Test_setConfigNewSuccess:PASS
library/component_types/Counter/test/Counter_Test.c:272:Counter_Test_setConfigDefaultSuccess:PASS

-----------------------
9 Tests 0 Failures 0 Ignored
OK

You may notice that the board performs a reboot after running the tests. This happens because the unit test binary has completed execution. Following the reboot, the failsafe image is then booted into since the unit test image will have been marked as unstable during its own boot process. This is expected behaviour.

If however, you chose to build the unit test binary for the failsafe slot, and programmed it to that slot, the unit tests would instead be executed repeatedly in an infinite loop since the image would continue to be rebooted, and the tests rerun, after each execution.

4. Troubleshooting

4.1. Faults

If you experience faults when running code on the Kryten, the information printed can help locate where the issue lies. Always include any fault output when contacting Bright Ascension support.

4.2. Wiping Corrupted MRAM

ACS run factory tests on each Kryten before it is shipped out. These tests can sometimes leave the MRAM corrupted. Corrupted MRAM will cause the memory controller to raise an exception within the CPU. This will result in the board boot looping. It’s possible to tell that this is the cause of boot looping by observing the debug output. In this case the debug output will show the EDAC MRAM error count to be non-zero. For example:

...
Flash EDAC non-correctable errors = 0
MRAM EDAC correctable errors = 12 (last at 0xA00FFFE0)
MRAM EDAC non-correctable errors = 36 (last at 0xA00FFFE4)
...

If this occurs, you can attempt to clear the errors by erasing the board’s MRAM. This can be done using the programming script, as follows:

hfk-kryten-program erase_mram