Serial ports still make the world go ’round whether you believe it or not. Serial ports were created on the second or third day depending on the translation of Genesis that you are reading. Serial ports can be fun little wires that connect one micro-controller to another micro-controller, and let’s talk a little bit about them.
Back in the good old days, every single computer in the world had one or even two RS232 ports. Why? So you could connect to that modem. So you could connect two PCs together to transfer files (since it was faster than floppy disks). You get bonus points if you have ever had to transfer data between computers using a null modem connector (pin 2 – pin 3 crossover) and just serial ports because nobody had network cards in those days. Oh, those were the days!
Why are we talking about serial ports you may ask? Well, because even before there was OPP hardware, there were calculations on serial bandwidth and would it be enough. There is no reason to start a project unless you first answer a couple of feasibility sanity questions and make sure what you are trying to do makes some semblance of sense. (This is why the propane powered snow melter for driveways never made it beyond the conceptualizing stage. The thermodynamics for that project didn’t pass muster).
The original Gen1 cards (using an HCS08 processor) ran serial communications at 38,400 bps. They were cute little cards that could support 16 inputs, or 8 solenoids with 8 inputs. There was no mix and match of capabilities. You could not make a 4 solenoid Gen1 card because, that card would have needed a different layout that I never created. At 38,400 bps, the cards could send and receive 3,840 bytes/sec. That doesn’t sound like much, but assuming polling every 10 ms, there were 38.4 bytes to send/receive data. To receive data an input card, the message would be 5 bytes in length. To receive data from a solenoid card (i.e. read the switch states), the message would be 4 bytes in length. That means that there is enough time to service a 4 input card + 4 solenoid card system. That would means there are 96 total inputs ((4 * 16) + (4 * 8)) for the system which seemed more than adequate to me. I didn’t really imagine putting 32 solenoids in a system and thought that could easily be dropped to 24 solenoids or lower which gave me more breathing room on the serial port. (Only 32 bytes per 10 ms).
So that design was assuming that I would poll the inputs every 10 ms. 10 ms seems pretty slow, right? Wouldn’t I need to poll the inputs faster than that? Well, how I figured it out was that I probed a pinball machine and looked at the rate of the column strobe for the switch matrix. It is now a little simpler, and I’ll instead refer to online information. According to the PinballRehab website, the switch matrix is strobed every 2 ms (500 times/sec), so if there are 8 columns, a read for each input is made every 16 ms. What you say? The processor only reads the inputs once every 16 ms or 62.5 times per second? Sorry that is it. Of course, the OPP Gen1 boards sampled the inputs much faster, did debouncing in software, and could detect and report on edges so even if the input wasn’t polled fast enough, it could “hold” that an input had occurred until the next read command came in.
So when Joe proposed the Gen2 boards, things changed slightly. Since the processor moved to a Cortex-M0 based processor, the boards could now support 32 inputs on a board, or 16 solenoids + 16 inputs per board. (Basically a doubling of the density). The price per system dropped further because less boards needed to be purchased, wing cards could now be mixed and matched so the exact correct board could be located at the correct location, and the serial speed went up to 115,200 bps or a 3x increase in speed. The great part is that this change in technologies did not change any of the physics of the game and how quickly inputs had to be read. Inputs were still occurring at the same rate, so…this just gave more head room. With Gen1 cards and the serial port, I felt I was a little too close to the edge of not having enough bandwidth. Yeah, SharpeShooter 3 worked perfectly, but that isn’t the most full featured game. With a 3x speed increase, I felt really good about the bandwidth on the serial port.
The Gen2 processor boards came with a USB to serial port converter chip on them. That now eliminated the need to have an RS232 electrical level to 5V level converter. Since some newer computers don’t have serial ports, it would have been necessary to buy a USB to serial converter and then convert the RS232 to 5V levels which seemed like too many adapters in my opinion. (Most modern SOCs do have UARTs built into them. This includes the RaspPi, BeagleBone, etc.)
So why am I talking about serial ports? It basically stems from the fact that the default MPF configuration tries to request information on the serial bus too quickly. (Maybe the default is to poll every 1 ms. I believe Jan has reduced the default polling rate for OPP, but I’m not 100% certain) Old versions of MPF requested information so quickly that it saturated the serial link and eventually data is lost on the serial link which causes data byte errors. The original plan was that the link would never be fully utilized…but what if.
As with all errors, I wanted to see where the errors occurred if the link was completely saturated. The serial communications goes from a host computer through a USB to serial converter, to a 115,200 bps serial link to the PSOC 4200, and then back again. USB ports are significantly faster than the serial link, so I wanted to make sure that the USB was throttled properly.
For the first test, I started by setting up a quick Python serial test to send a large amount of data to USB virtual COM port and wrap it back without going to the PSOC 4200 processor. That just tests the USB to serial port hardware and driver. I threw down 260K of data, and examined all of the data coming back to verify it was correct. The test insures backpressure is properly happening when going from the faster USB to slower serial port. I also watched the output buffer, and verified that it was backing up because of the data going into the slower pipe. That test worked 100%, which proves that the backpressure works and the buffering isn’t limited by a hardware buffer inside the USB to serial port chip.
The next test that I ran sent all of the data to the PSOC 4200, but used a card address that doesn’t match the current card. That means that all the data simply passes through the card without processing the information. I once again threw 260K of data, and examined the number of bytes received. I got a .74% loss rate on the data. Examining the data, I also found that I would lose a single byte at a time, and was not dropping large chunks of data and recovering. That was good news because it helped prove my theory that it was the internal buffer in the PSOC 4200 that was overflowing and losing data.
So why is saturating the link so bad? It means that as commands are sent to the card, they are buffered in internal memory for longer and longer before actually getting sent to the hardware. A quick analogy might help. Let’s say that you have a sink that the faucet can fill faster than it can drain. When you start, everything is great because the extra water just gets added to the sink which is the buffer. But as it keeps filling, eventually it overflows and gets all over the floor. (That of course ignores the overflow drain at the top of the sink.) With a computer, it is even worse. The commands being put into the system are buffered in a FIFO fashion (First In First Out). As the buffer fills up more and more, the commands take longer and longer to get to the hardware, and suddenly when you tell a solenoid to kick, it takes 1/2 second or a whole second before that solenoid finally gets that command. As your system runs longer, commands take longer and longer to occur since the buffer keeps getting larger and larger. That is very bad. Your system becomes much less responsive.
When first discussing this problem we batted around the idea of adding Xon/Xoff flow control. There are two problems with that. First problem is that it would not stop the buffer from continuously increasing in size and making the system less responsive. (It would actually exacerbate that problem). Second problem is that OPP commands are binary and as such the Xon/Xoff must be “escaped” to insure that data that is the Xon character can be distinguished from the actual Xon character. This ends up meaning the amount of data needs to increase to add these escaped sequences. (I believe that Fast and PROC both use text based commands. Xon/Xoff are non-printable characters, so because of that they do not need to escape the characters. The downside of doing that is that the protocol is much less efficient on the wire. I.e. when they send a byte of data, they send two ASCII characters to represents the bits. To send a byte of ones, they would send two ‘F’ characters or in binary 0x46 0x46. OPP simply sends 0xff and is done with it. It means that the OPP protocol can be twice as efficient on the wire. (The down side is that it is much less human readable. I can’t just look at a string of characters going past and easily tell what is going on because many of them are unprintable characters.)
So while doing testing, I learned a couple more things about the buffering of the Cypress USB to serial port. In python you can ask the serial port how many bytes are waiting to be transmitted (out_waiting parameter). When making sure that this value is 0 (i.e. the buffer is empty), there are still up to 128 bytes of data in an internal buffer in the USB to serial port chip. That is very problematic because you can’t assume that there is an idle time on the line by polling the out_waiting parameter. (You need to wait for out_waiting to be 0, and then wait another 10 ms for the USB to serial port chip buffer to clear). I also found that the serial link could handle bursts of 2160 characters without losing any characters. That shows the amount of time it takes for the slips on the serial link to build up and eventually overflow the PSoC 4200 SCB buffers.
So long story short, what is the best fix for this? I feel adding Xon/Xoff doesn’t correct the problem, and could lead to an even less responsive system. (And it isn’t just because I’m too lazy to add Xon/Xoff and escaping to the protocol. It just isn’t the correct fix). I really believe that tuning the polling rate is the correct thing to do here. I’ll look through the OPP MPF platform and see if there is an easy way to auto-tune the rate so others don’t need all this knowledge to make the right call on how fast to poll the hardware. I gotta get in there anyway to add OPP RGB LED support.
Open Pinball Project 