A Real Time Scheduler for Digilent MAX32

Introduction

The Digilent MAX32 is a powerful embedded board, based on the Microchip PIC32MX795F512L, a microcontroller with 64k RAM, 512k Flash ROM, and lots of IO. It was designed as a kind of PIC-based Arduino - but, with a USB serial port, on board voltage regulators, and a header for bare-metal programming with Microchip's free toolchain, it is a great platform for general embedded projects. In this article I show a real time scheduler, with debug printing, and error handling which works well for hard real time work.

(The scheduler in this article is based on RIOS, which I already discussed in a previous article. If you are new to this topic, I suggest that article as a starter.)

Design of the debug print turns out to require writing some code with a critical section (so that different interrupt levels can all write safely to the same print buffer). So we also look at some test code which shows this feature working (and not working).

Digilent MAX32 running on the bench

GETTING STARTED

Code for this article is available from my Github account, along with the necessary project files. If you have MPLAB X IDE installed and clone the git repo, you should be able to build this project and use it to program the MAX32.

This scheduler is designed to be as simple as possible, while still giving you all you need to build a hard real time system. It can be modified and hacked without too much hassle. (I call this approach DWINTDANM - Does What It Needs To Do And No More).

FEATURE LIST:

SYSTEM:

SCHEDULER:

DEBUG PRINTING:

ERROR HANDLING:

TEST/DEMO MODE:

DEBUG PRINTING DESIGN

One of the issues I hit using the RIOS scheduler was debug printing. (I consider a decent debug print method to be almost essential on a serious project. I know that there are other ways to do things, and I use them if I have to, but really I want to just throw "printf()" statements into my code.)

But there is a problem. A UART takes time to write a character - appreciable time. Our debug port runs at 921.6kBaud - for an 8N1 print, that's almost 11us per character. If you are writing code that is trying to meet tight timing specs, you just can't block for that long.

BUFFERED DEBUG

The cooperative scheduler always has some dead time at the end of each timer tick, so my solution is to use it. Instead of writing directly to UART, "user code" just puts characters directly into a ring buffer. During the "dead time", any characters in the buffer are fed to the UART. So long as we are not too profligate with our printing, and system load is not too heavy we can have predictable RT performance, and a debug UART.

CRITICAL SECTION

But there is another possible problem. I want to be able to write to debug from ANYWHERE in my code. The task scheduler runs at interrupt level 1 (lowest) and as everything happens sequentuially, there is no issue with writing the buffer here. But if a higher level ISR wants to write to debug, I want it to work. (I don't leave this in my code, but I might do it temporarily to check an ISR is running and is seeing the data I expect.)

Now we have a classic problem : the print buffer has multiple writers, which can try to read-modify-write the buffer indexes at the same time. We need a critical section. This can be done in several ways; one is to turn off interrupts for a few instructions where the index is updated. But that's not great - we might have high priority interrupts which are time-critical.

It turns out though that the MIPS core used by the PIC has this taken care of. There are special instructions, Load Link and Store Conditional (LL/SC) which handle exactly this. And the GCC compiler has a set of instructions designed to use them - the one used here is __sync_fetch_and_add().

You can find all this in the file debug_uart.c.

TESTING THE CODE:

I really wanted some way to test that this design was working as I expected. This meant first writing some test code and triggering problems when the critical section wasn't there. To do this I added two test tasks to the design - one to write to the debug buffer from the task scheduler, the other to trigger a timer which in turn fires an ISR, interrupting the buffer writes from the first task. This code is included.

If you grab the code from github, you should be able to build and program the MAX32. Now, if you attach the MAX32 to your PC with a USB cable and fire up a terminal program (I am using Realterm) you should be able to connect to the board at 921600bauds. (You may have to fiddle around to find the right serial port.) Then you should see something like this, after hitting reset:

Digilent MAX32 debug print in test/demo mode

Obviously the first two lines are just an initial printout to the debug port. The other lines are from the test tasks. (You can find them all in the file scheduler.c.)

(We can see here that debug prints from higher levels share the buffer at character level. Strings from different levels can and will get jumbled up; but this is fine for debugging. It probably is possible to arrange that complete strings get written - but I didn't think it worth doing. Maybe I will one day if it becomes a nuisance.)

TESTING THE TESTS

This is all good, but how do we know the tests are real? We need to see the buffer getting corrupted (i.e. characters getting omitted.)

To do this, go to file debug_uart.c and comment out #define CRITICAL_SECTION_SYNC. Re build the project, and upload to the target. Now you should see something like this in your terminal:

Digilent MAX32 debug print in test/demo mode with errors

What happened here? You can see that the ISR debug print for count 14 got completely clobbered by the lower level task. If you look closely you will probably see the odd missing character as well.

If you want to you can also try implementing the critical section by turning interrupts off. The C macros are already there, in debug_buf_put() function, also in debug_uart.c. If you uncomment them, rebuild and rerun, you will see the errors are gone.

ADDING YOUR OWN TASKS

When you are done, uncomment the #define CRITICAL_SECTION_SYNC at the top of debug_uart.c and rebuild. To remove the test code, you can comment out the line #define INCLUDE_TEST_TASKS at the top of scheduler.c - or just remove the test code.

You can now start adding your own tasks in scheduler.c, using xprintf() as you go to test and verify.

ERROR HANDLING

We have a simple and quite uncompromising way to handle errors. The function fatal_error() takes two argument - one is an error number, one is a an error string. It completely halts ths system (it actually turns off interrupts), and enters a hard-coded loop which prints an error message to debug every 5 seconds. The RUN led turns on solid. If this should ever happen in your system, you'll know about it, and you'll get some idea why if you attach a terminal - even if the terminal wasn't attached when the error happened.

LOAD MONITORING

The RUN LED also functions as a simple visual load monitor. The heavier the (worst case) load, the long the LED is on for each second. This is done by updating a system variable system_timer_max (it has a minimum value of 100 so we always see at least a short flash). This is done in task_load_monitor(), which must run as the last task. This value is then divided and used to set the on period of the LED in tasks task_blink_on() and task_blink_off(). All of this is in scheduler.c.

SYSTEM TIMING

By default, the PIC runs at 48MHz, debug at 921.6kb, and the system tick is at 5ms. None of this is fixed in stone - you could change the clock and/or the tick if you want. the tick is fairly easy, but if you change the clock from 48MHz, you will need to adjust pretty much the whole system timing - baud rates, timer periods, and so on.

THAT'S IT

I hope you find this code useful and perhaps educational as well. Comments and so on are welcome via the contact page.