An exploration of the hardware in a Cisco 2503 router.
It turned out that while I was working on this, so was somebody else [1]. They've done a great job on documentation and there's little point replicating the work unless I see any differences that need mentioning. You will find in this repo that there is an emulator of sorts that was created to help understand the existing code and test new software. A monitor that was developed to test that understanding with the real hardware, and there's also a U-Boot port with support for the serial console (obviously), system timer, NVRAM, boot flash, and ethernet port.
This project started out because I was looking for some hardware which had a Motorola 68k processor for experimentation and as a refresher of 68k assembler. Having looked at various devices I decided to reject the usual crowd of Amiga/Apple/Atari/etc as too expensive, and requiring addition equipment (ie displays). I wanted something standalone, a case and PSU being an advantage. Continuing the search I came across the Cisco 2500 [2] line of routers which are readily obtainable from Ebay and are modestly priced. The wiki page reveals that this is based on a Motorola 68EC030 which is the cost reduced variant with no MMU which is disappointing, but otherwise the specs are reasonable. Looking at the picture on the wiki site, a couple of important points come to light which are that even though this is an 'embedded' device the boot ROMs, main flash, and RAM are all pluggable (and therefore easily replaceable), and that there is no expansion bus which is no surprise. Some more Googling reveals some work had already been done as there is a uCLinux variant available [3].
I ended up purchasing the 2503 variant which is the base model with an ISDN interface, this was for no other reason than it was cheap though this may prove handy in the end (read on...). The main components on the board are:
- CPU: Motorola 68EC030
- RAM: 72 pin SIMM DRAM (upto 16MB)
- Boot ROM: 2 x 32 pin PLCC PROMs
- Main ROM: 2 x sockets taking 80 pin SIMM with parallel flash ROMs (AM29F016B)
- NVRAM: 32k EEPROM for router configuration
- 2 x large ASICs with Cisco part numbers
- Serial: Philips SCN2681 dual UART (console/aux RS232 ports)
- Sync. Serial: Hitachi HD64570 serial communication adapter
- Ethernet: AMD am79C90 "LANCE" ethernet controller
- ISDN: TP3420A ISDN interface device (Cisco 2503)
- ISDN controller: Motorola 68302 (Cisco 2503)
That last part is quite curious as it's a SOC (System-On-Chip). It contains a mode selectable 68000/68008 core with a full suite of additional peripherals including GPIO and a number of configurable serial peripherals supporting different protocols. It also contains a secondary RISC processor that can be programmed to automate external communication using the SOC's peripherals. The final curious detail is that the SOC's 68k CPU core can be disable (as in this case), and the unit used as a peripheral to a standard 68k CPU, nifty.
So the first job was to pull the boot ROMs, and dump the contents. The 68030, while a 32 bit (internal/external) processor supports different bus widths when reading different devices (see DSACK0/DSACK1 signals), and the 8 bit boot ROMs form a 16 bit wide data space. So while the ROMs are keyed to avoid insertion in the sockets with the wrong orientation, they must be put back in the correct socket or it won't boot. The first thing I tend to do when doing a project like this is take a number of board shots for reference purposes for precisely this kind of point, and it also helps that you can use them to try and identify the various components on a board. The ROMs were dumped with my trusty Dataman programmer and a PLCC to DIP convertor. Swapping back to Linux (programmer's Windows only) I tried recombining the ROM dumps but either way round (the ROMs represent odd and even addresses, so need to be interleaved) the data didn't make any sense. Opening in Radare2 (a reverse engineering tool that includes a disassembler for 68000) didn't recognise anything, 'strings' (which as the name suggests dumps text strings from a binary) also didn't find anything which was strange. I started to wonder if I'd selected the wrong device when I'd dumped the ROMs, and had the address lines muddled (the data was changing suggesting the data lines were at least correct). I also considered that the ROMS might be encrypted, the large ASICs could potentially hold a decryption engine, or that Cisco might have reversed the data bus (ROM D0 = CPU D15, ROM D1 = CPU D14, etc) to throw people off. I decided the best way to proceed was to just 'buzz' out the circuit using a continuity tester, the advantage of working on older equipment is no BGA (Ball Grid Array) like devices so all the pins are available. To complicate this process Cisco decided to not mark the components with a silkscreen so I ended up creating a board layout with arbitary component numbers just to facilitate this process. With this I mapped out the various pin connections of the ROM sockets to any intermediate components (74HC series logic) to it's destination ('notes/U3_U4_ROM.txt'). I prefer to do this 'blind' to start with, that is to say not look up component datasheets, just to avoid any preconceptions. Having done this the datasheets were found for the 74HC logic devices which turned out to be buffers (no surprise), so at this point it can be seen that that the ROMs are connected to the CPU address/data buses by buffers, all quite standard, no encryption. The next job was to map out the actually address/data bus paths for each signal to determine how the ROMs are actually connected to the CPU. Starting with the address bus it was immediately apparent that something was a little off, literally, each signal was off by one (ROM A0->CPU A1, ROM A1->CPU A2, etc) and there was no connection to CPU A0. It took a minute to realise that I was being an idiot and of course there's no connection to A0 as the two ROMS represent the two different states of A0, the original 68000 CPU doesn't even have an A0 pin (though it does have signals to differentiate the two states). This comes back to preconceptions, if you expect to see a problem you see a problem. Moving on to the data bus, things get more interesting. From the start it was obvious that something was not right and tracing the signals it was apparent that Cisco had indeed reversed the data bus (CPU D0->ROM D15, CPU D1->ROM D14,......,CPU D14->ROM D1, CPU D15->ROM D0). I cursed myself for not trying what I had already thought of several hours earlier. To be fair, my reckoning was that if you are going to go to the trouble of doing this then you might as well remap the data bus by randomising it, it wouldn't make any differnce to code execution and make ROM image generation only slightly more complicated. Looking from the otherside maybe Cisco thought this was a simple way to slow down reverse engineering, and it wouldn't matter how they wired the data bus if people were willing to 'buzz' it out. With this knowledge the ROM images have their bytes values reversed, and then interleaving them correctly produces a ROM image which not only produces some interesting text using 'strings', but also Radare2 identifies as 68k code!
Part of the purpose of this project was to try and turn the router in to a standalone 68k development platform, and to do this requires a (much) better understanding of the internal configuration of the hardware. While some of this information can be gleaned from physically examining the PCB to see how components are connected together, the major difficulty here is that there are two large ASICs (Application Specific Intergrated Circuit). These obviously play a major part in the operation of the device as they are connected directly to the CPU bus and connect to the CE (Chip Enable) lines of pretty much eveything else. On top of that they also appear to be an interrupt controller, and are responsible for generating the correct bus signals when accessing different memories/peripherals. Obviously Cisco has never made this information public, so the only way to try and understand these I.C.s is to examine the boot ROMs (and potentially main flash). Doing this in Radare2 is posible but cumbersome, this is not a dig at that project, I found it extremely useful on a number of occassions. It is however really quite complex, with a TUI (text user interface) which doesn't help. Some progress was made, with one of the most important register bits documented this way, it swaps out the boot ROM which occupies 0x0000 0000 on boot with RAM. This was hardly elusive though, it's almost the first thing the boot code does. So to try to speed up the process of understanding the boot code execution an emulator seemed a possible way forward. I wasn't interested in trying to write the CPU core as there are a number of open source projects available that can emulate a 68030, so looking at the capabilites of these projects I decide to go with the 'Musashi' core due to maturity and expandability. Having used ncurses (terminal TUI interface library) before, I decided to implement the user interface with that. To start with the process was simple enough, step through the boot code until it hits something it doesn't know about. At this point the emulator was being updated with basic peripherals (RAM/ROM) that were already known about thanks to, at least in part, the documentation from the uCLinux port. The TUI was also being improved with better instrumentation and execution control. It has to be said this was written to provide an insight in to the boot code, not as a model of good coding practice! One of the peripherals that was added early was the DUART controller. Having implemented the peripheral's core logic I could connect some terminal software (minicom) to the emulator using 'socat' (see, 'serial/socat_serial_emulation.sh'). It was quite satisfying to see the Cisco boot message being (slowly) printed to the terminal console. While 'socat' is good enough to emulate a basic serial link, to test some of the more complex hardware initialisation I wrote a kernel module which creates pairs of linked serial devices.
At this juncture I had enough information to attempt to create a monitor for the device, so development of a monitor began in tandem with the development of the emulator.
I had recently written a monitor for a Z80 platform [4], so decided to reuse the design for this one. Having the emulator available made development easy as code could be tested without needing to burn a ROM, and the internal state was readily available making debugging a breeze. It was soon time to test the code on some actual hardware. While working out earlier how the ROMs were connected, an interesting thing came to light which is that a couple of the socket pins can be reconfigured by the jumper block to the left (from the front) of the sockets. Looking at what this would do if the plugs were moved to the alternate position it became clear that the board could support flash ROMs. This was an excellent, if unexpected, discovery. Back to Ebay, and couple of devices were sourced. It was about this point I added the ability of the emulator to read the original ROM images as they were dumped (ie. reversed), so that I could check that the monitor ROM images I wanted to burn. You can use 'srec_cat' to split the assembled firmware file in to odd/even byte images, then there is a small python script 'bin/flip_bits.py', to reverse each individual byte (see, 'monitor/Makefile' for more info). When the flash ROMs arrived, the programmer was fired up and devices burnt. Taking care to plug them in the correct socket, I powered up the router and was rewarded with the monitor interface.
So at this point I had an approximation of an emulator and a basic monitor on the device. From here I could use the monitor to explore the function of the system registers which the emulator revealed and try and deduce their function. This was easier in certain cases than it was in others. The system registers controlling the RAM/ROM sizing were easy enough, and their effect quite measurable. They control what's regarded as a valid address access for those devices, outside of which a bus error will occur. One of the more annoying registers is actually the interface to another memory device that I'd missed previously. Connected to one of the ASICs is a 24C44, which is 256 bit (yes, that's correct, there's no prefix) static RAM device backed by EEPROM store which on power-on loads the RAM from ROM. This is used to hold non-volatile information such as device model (the firmware is meant for several different models, and not just 2500 series), ethernet MAC, etc. The 24C44 is a syncronous serial device which is SPI like in having an interface comprising CE (Chip Enable), DI (data in), DO (data out) and CLK (clock). The system register was just an I/O port to those lines, driving the CE, DI and CLK, and reading the state of DO. In trying to workout what that was, I ended up implementing an additional feature in the emulator which works in tandem with a program running on the monitor which passes any read/write requests to unknown addresses in the emulator to the actual hardware. Once I could actually grab a log of what was actually being executed, and what was being saved, it was easy to deduce what was happening and why. This method of interfacing the emulator to the real hardware proved handy on a number of occassion, but it did come with the cost of being really quite slow (not surprising though running over a serial link). One place it didn't help though is towards the end of the initialisation process where it performs an initialisation of the RAM, before going off and executing the main flash ROM. Executing this on the emulator didn't work because there was a check of one of the system registers and it wasn't being set correctly. Executing this in conjunction with the real hardware resulted in the hardware reseting and a confused emulator as the device had stopped communicating. Examining that section of code more closely, it's apparent that it updates the bus error exception handler, so if it is expecting that to fire no wonder the real hardware reset. Having updated the tools to handle that exception I tried again. This time the hardware didn't crash, but the emulator didn't move past the check either. Now it's at this point I really wish I'd being paying more attention when I'd run 'strings' across the boot ROM image, because though out this I have been deducing what the code is doing from the effect on the hardware, if only there was a little more information about what the code was doing things would have been faster... Well turns out there is, doh! I was digging through the code and came across some strings such as 'Testing low memory', 'Enabling timers', and 'Enabling Watchdog timer'. Well Googling revealed that there is a diagnostics mode that you can enable by changing a bit in the NVRAM configuration. So after updating the NVRAM, trying again rewarded me with additional information being display, and it printed just before the offending block of code... 'Parity Logic tests'. It's running a memory parity check, so no wonder the hardware doesn't help here as the memory operations are never carried out there. Well, that's that one solved. Another configuration bit which can be set, boots Cisco's monitor ROM which works with no problem (up until you try to do something with unemulated hardware). I used the monitor to make a copy of the main flash ROM, and implemented that in the emulator. It can boot a bit of that, but it soon runs into parts which aren't emulated. It was however quite helpful to see how it was initialising the ethernet controller and the 68302.
And now there is enough information to attempt the next stage.
I have used U-Boot on a number of different boards, compiled it more than a few times and even done a relatively simple port for a Wifi AP [5] (this mostly involved reusing exisiting components rather than writing new code). I felt the next challenge should be a port to a board that is unsupported as possible, and this fits the bill. By bring U-Boot to this system I can get basic IP support on the ethernet interface, rather than having to implement a driver/IP stack/tools in my monitor.
Starting with a fresh pull from the repo [6], the first job is to create a CPU definition. While none of the 680x0 processors are supported, some of the ColdFire devices are supported by U-boot. ColdFire devices are 68k derivatives incorporated as SOC, and while they are not totally code compatible any basic code written for the ColdFire series will run on the 68k series. So one of the existing CPU configurations was copied and updated for the 68030. This included removing any ColdFire specific code and adding an early board initialisation routine to switch in RAM before any stack is used. Basic board, config, devicetree, and include files were created, again using previous examples. Then the various build files Kconfig/Makefiles were updated and a test 'make' run to ensure the new files were being included/built. Next a driver was written for the serial console, and intergrated in to the build files/devicetree. The initially build of U-Boot was run using the emulator and once again it proved most helpful working through a couple of minor bugs. It has to be said that resolving bugs in the early stage before the console is available can be challenging, devicetree can be a pain in this regard. A driver was then implemented for the NVRAM to provide U-Boot with some environmental storage. Next was the system timer, U-boot is very dependent on having a timer available as a lot of functions/commands use a time-out. After that support for the boot ROMs was then added. This was a little tricky because though U-Boot has drivers for parallel flash devices, the fact that Cisco reversed the data buses means the flash management commands/results are invalid. With this complete a fully self hosting U-Boot environment is available.
The last major peripheral to be added was the ethernet controller as it was the most complex, and least needed. Initial code development was done using U-Boot's standalone API, as this allowed code to be tested without overwriting the boot ROMs. The ethernet controller is quite simple in having only a few registers itself which are used for basic control, status and bus configuration (as it supports a variety of different CPUs). There is also an additional register pair that points to a section of memory which holds the rest of the device configuration. While I'd managed to find the datasheet for the device I was having problems trying to get the device to initialise correcly. I had even tried copying the configuration data from the original ROMS with no success. It was only when I copied the entire initialisation routine verbatim from the original firmware that the device responded correctly. This left me scratching my head for a little while trying to determine what was actually different about the original code. This is when it occured to me that the configuration block of the device was being created in low memory in my code, where as originally it was being created in high memory. Experimenting with where the configuration block was located confirmed that the device fails to initialise if the the block is located below 2MB (out of 4MB of RAM). Subsequent research [1], shows that there is I/O window (enforced no doubt by one of the ASICs) which blocks the LANCE accessing memory below RAM size - 2MB. Having developed the test code, I could then proceed to write a driver which was mostly uneventful. It's ironic that if I'd developed the code as a driver from the start I'd never have run into this problem as U-Boot loads itself into high memory. Having sourced an AUI to twisted pair media converter (from Ebay once again), it was time to try and see if it would work. As always with these things, not on the first attempt. Trying to ping another host resulted in a packet alledgely being sent (according to the Tx LED on the media convertor), but no response. Firing up a packet sniffer didn't help, couldn't see any packets from U-Boot. Looking at the switch seemed to indicate something transiting, but then why nothing in the packet sniffer. Well one of the nice things about Linux is that it tends to be quite verbose, and looking at the port statistics the error count was increasing, so something was being received. Further investigation revealed that this was due to the packet being to short. Refering back to the datasheet it can be seen that the LANCE does not pad packets to the minimum ethernet frame size, so having fixed this I was rewarded with a working ethernet interface.
Looking at potential future modifications there are a couple of possiblities. In regards to hardware, the storage capacity of the main flash SIMM sticks are okay, but access to some removal storage would be advantagous. Here's where the 2503 could come in handy, the ISDN interface is connected to the 68302 by SPI. This could be leveraged to provide a SD card interface, requiring just the SPI lines DI, DO, CLK from ISDN interface and an additional CE which can almost certainly be sourced from the 68302. Another potential avenue of expansion is at the back of the board tucked behind the flash SIMMS. Here you'll find 4 rows of 17 holes which look suspiously like it could be for a PCMCIA socket, which would make sense as some models shipped with the boot ROM on a PCMCIA card. In addition there is the jumper below the Boot ROMs which has the CE line in the center, and as connected provides it to the ROM sockets. I didn't manage to trace the other pin originally as I wasn't think about that area, but it does seem likely to lead there. To use this in addition to the boot ROMs would require a chip enable line to be fashioned somehow. If this was put in as an addition, the other bus signals would have to managed some how, however if it was hijacked from another missing peripheral, that peripherals address space could be used. One potential source could be one of the flash SIMM's CE, and the memory hole for the SIMM is certainly large enough to accomodate a PCMCIA card. The problem with that though is that the flash SIMM's are 32 bits wide, and there's probably no way to alter that. If you wanted to just add some additional peripherals, this is not a problem, the address space would be broken up but still usable. With memory this would be less helpful.
As far as software goes, there already is a FreeRTOS port [1]. Linux is probably not possible, RAM is too limited and CPU lacks a MMU. CP/M-68K is a possiblity depending on the availability of software.