Let’s get some history: Jeremy is a long time supporter of OPP (Open Pinball Project). He was one of the original backers of the OPP kickstarter. He runs the Pinball Makers website and created many of the instructions on how to build and connect together OPP cards. I had documentation, but Jeremy had much better skills at putting together better diagrams to make populating the boards more easily understood by a person other than me. Jeremy rocks!, and I thank him for all of the hard work that he has done over the past three or four years.
Jeremy decided to switch his pinball controllers from OPP to PROC (I really have no idea what the PROC acronym means). I support anyone’s decision to choose the PROC because it is a very mature platform. I think of OPP providing a service for hackers mostly. I think of PROC as a full fledged product. Side note: I was talking to Gerry a couple years back, and I believe he thought OPP was stealing his customers. I told him that I believe that OPP would eventually drive customers to his products. I simply think this is what happened in this case.
So nobody changes their pinball controller without having a heck of a good reason. A major change like that involves a significant amount of work for rewiring, and a lot of extra cost for buying those more expensive controllers.
I have rethemed and run two different machines using OPP controllers. They work without issues, and my main complaint is that their boot time is too long. Why are they different than Jeremy’s machine? The main difference is that I use the OPP pinball framework versus using MPF (Mission Pinball Framework). This points to the Achille’s tendon of OPP boards. The communication to the boards are slower than all other pinball controllers out there.
Let’s give a bit of history on OPP. It was meant to control late ’80s style pinball machines. That was the goal. Not ’90s style with DMDs and all sorts of bells and whistles, but the much simpler ’80s style with incandescent bulbs and such. This was mostly based on my knowledge of how much skill I had at making a pinball machine. I think that I looked at both Firepower 2 and Dolly Parton as a realistic goal of what I needed to be able to control. Firepower has a lamp matrix so I used that as a goal for what OPP boards needed to support and how frequently updates were required to lamps.
After a little research on lamp matrices, each column is strobed for 2 ms, there are 8 columns, so each light bulb is updated every 16 ms. An insert light that is “on” is really only on for 2 ms out of 16 ms, but because of persistence of vision, it appears to be on all the time. Since OPP bulbs are on 100% of the time, there was really no reason to update lamps more quickly than once every 16 ms, and even that, doesn’t make much sense because it will look like a bulb is continuously on.
Another typical mode for a bulb is to blink either rapidly or slowly. From previous projects, the most rapid blinking that is comfortable to me, and doesn’t just look like flickering is about 100 ms. A slow blink is about once per second. Since that is needed so frequently in pinball, (i.e. blink an insert rapidly or slowly) I created OPP commands so that the PSOC 4200 could blink at either fast or slow speeds automatically. That further reduces the need for serial commands and reduces the need for updates to less than every 100 ms for each board.
The OPP framework uses the above types of commands to minimize serial communication. How it is actually implemented is it uses a 100 ms tick and collects all updates to lamps during that tick. At the end of 100 ms, the framework gets what the current state of the incandescent bulbs should be, and sends a single command to update them if they are different than the previously saved state.
On MPF, I tried to implement the same sort of controls (without using the blink commands since there is really no equivalent in MPF). I thought I got pretty close, but after looking through Jeremy’s log files, I can definitely see that I didn’t achieve what I wanted. (Since I don’t have an MPF machine, most of the coding that I do in MPF is “blind” coding. I don’t have a good way to test to code, and really rely on others to make sure it is correct.)
MPF seems to use an individual lamp as an object. (Object oriented wise, that makes a ton of sense). The OPP pinball framework uses the whole card as an object. I organized it that way to make it easier to collect updates to all lamps on a whole PSOC 4200 to minimize the number of serial commands. The MPF object view makes aggregation more difficult since individual lamps aren’t tied together into a single object. I tried to get this to happen but evidently I didn’t do a good job.
One last reason to minimize turning lamps on and off quickly is flyback voltage. Jeremy kept reporting that the incandescent boards were getting really hot from the BS-170 MOSFETs. That confused me because my MOSFETs didn’t have that issue. (Van Halen uses BS170s just like his machine). What I failed to take into account is that the framework was turning the MOSFETs on and off very quickly. Just like with solenoids, when an incandescent MOSFET is turned off, the flyback voltage builds up. With solenoids, an extra diode is placed across the solenoid to bleed off the voltage. With an incandescent bulb, the current is much lower so it doesn’t build up as much flyback voltage, so the diode isn’t normally necessary. But…if the incandescent is switched on and off rapidly, it becomes more of an issue, especially because of heat build up at the junction of semiconductor.
So here is an example from Jeremy’s log files, right about when everything goes wrong:
2018-05-23 06:30:44,861 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x00 0x0f 0x5f 0x36 0x23
2018-05-23 06:30:44,866 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0xdc 0x10 0x4e 0x0e 0xd6
2018-05-23 06:30:44,871 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0xdd 0x10 0x4e 0x0e 0xc0
2018-05-23 06:30:44,875 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9d 0x10 0x4e 0x0e 0x5b
2018-05-23 06:30:44,880 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9f 0x10 0x4e 0x0e 0x77
2018-05-23 06:30:44,884 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x10 0x4e 0x0e 0x2f
2018-05-23 06:30:44,889 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x10 0x5e 0x0e 0x78
2018-05-23 06:30:44,893 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x00 0x5e 0x0e 0xda
2018-05-23 06:30:44,898 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x00 0x0f 0x5f 0x37 0x24
2018-05-23 06:30:44,906 : DEBUG : OPP : Update incand cmd: 0x23 0x13 0x07 0x00 0x00 0x00 0x3f 0xf2
2018-05-23 06:30:44,910 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x84 0x89 0xf8 0x00 0xcd
2018-05-23 06:30:44,915 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x1b 0x00 0x5e 0x0e 0xeb
2018-05-23 06:30:44,919 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x1b 0x01 0x5e 0x0e 0x80
So within 58 ms (44.919 (end timestamp) – 44.861(start timestamp)), 13 update incandescent commands are sent. Jeremy had eight cards in his system, but they were broken into two strings of four cards each. The first string has two PSOC4200 cards with incandescent wings. The second string has a three PSOC4200 with incandescent wings. To completely update all the incandescent bulbs in the system, it should only take 5 commands. Anything more than 5 commands will not be perceived by the human eye because of persistence of vision. Again, I believed that I had implemented this in MPF properly, but I guess I did not. The second reason I believe this is the issue is that it is right when things failed in his machine.
What else went wrong. Well, Jeremy reported that he was seeing CRC errors, and when I originally looked into the log files, I couldn’t find any CRC errors. Because of that I kept emailing Jeremy asking for log files that showed the errors (and ignoring the log files that he sent me), while Jeremy was probably thinking to himself that the log files were showing the errors. Jeremy was just reporting what he had seen before. (a CRC error looks externally the same, but is really a very different beast). I assumed that I didn’t receive the correct log file, when I should have realized that the issue might have been described incorrectly. (Unfortunately, it was also at a very busy time in my real job which meant that I couldn’t spend as much time on it as I usually would)
So instead of communication issues, I should have been looking into recovery from a buffer overflow. Again, with the OPP pinball framework, the machine never gets buffer overflows because it uses a tick to insure that a buffer overflow can’t happen. Now, knowing that MPF can easily cause buffer overflows, I need to spend more time to make sure that the OPP firmware can recover gracefully from overflows.
The above discussion may be seen as me pointing the finger at somebody else. Far from it. MPF provided a ton of features that Jeremy utilized. MPF provides stock code for running the ball trough. (That is all individually coded in the OPP pinball framework and isn’t a nice simple object.) MPF has stock code for running EM style score reels. (Score reels require a good amount of code to work properly, and it is definitely not something that the OPP framework would support without hand coding it.) The OPP framework is straight python code, while MPF offers yaml to make scripting rules easier. That helps people fearful of coding feel more comfortable. The biggest thing MPF offers is the graphics capabilities. Look at the Nightmare Before Christmas pinball machine and see what Mark was able to accomplish using the MPF framework. (The OPP framework only really supports static images. When switching songs on the jukebox in Van Halen, there is a single image for each “high-lighted” song selection so it is really just switching the background image)
Well there it is. That is my retrospective on what I think went wrong.
Open Pinball Project 