12/30/2018 – Arduino, Knockout

I think this is my final Arduino entry.  I finished implementing and testing using two solenoid wing boards using an Arduino nano, and everything is working as it should.  I didn’t implement any of the serial communications mostly because I stopped understanding what I was gaining using the Arduino nano platform.

I chose the nano because of the price and it was approximately the same cost per pin as the PSOC 4200.  The PSOC is $4 an has 36 I/Os, and the Nano is about $2 and only has 16ish usable I/Os.  The PSOC is a much more powerful processor, so I was losing some performance there.

The one nice thing is that the Arduino has the super easy hardware abstraction layer (HAL) which makes it rather simple to program.  The problem was that every time that I used the HAL, the performance was atrocious because I didn’t like the decisions that were made to make it simple.  There are a ton of busy waits which makes programming easier, but it reduces the processor usefulness even further.  To get true real time performance I ended up bare metal programming the processor.  That meant that I really wasn’t using the Arduino platform at all, but I was just coding another inexpensive processor.

So then there are the pinout choices that the Arduino made.  It was very difficult to get a group of eight useful pins in a row which meant making very small wing cards work properly was a pain in the butt.  The Atmel choice to not make all of the analog inputs to be used as digital I/O further complicated the code.  (Atmel has said they will fix this in the next version of the chip).  While coding, I had to keep a map of the pins sitting beside me because the pinout was not logical.

All in all, I just don’t see what I was gaining.  That’s why I’m ending it.  The price difference was not sufficient to make it worth all the extra work.  The performance was significantly lower.  I enjoyed playing with the Arduinos, but at this point, I’m at been there, done that, stage.

Advertisements

11/17/2018 – Arduino, round 2

Well the Arduino nano performance was atrocious, but I blame that mostly on the Arduino framework itself.  After doing a little bit of research, it seems that reading the analog inputs occurs using a busy wait.  A read can take approximately 100us, so that means that most of the time is spent waiting for the ADC to complete its conversion.  That’s what I will target to make it more efficient.

So in my first rendition of the program, I read four analog inputs and four digital inputs for the inputs of the solenoids.  I could just as easily have read two analog inputs, and six digital inputs to minimize the number of analog reads.  The second program ArduinoBareOPP.ino uses the smaller amount of analog inputs to minimize those reads.

If the Arduino framework forces me to busy wait reading from the analog inputs, it is time to add an interrupt in.  The interrupt gets called after the ADC completes.  At that point, I set the mux to read from the next channel and kick that conversion off.  That means that the conversions are happening in the background without busy waiting for those to complete.  In the interrupt, I also fill currData which holds the value of the inputs.  This is also where I convert it from an ADC sample to a single digital bit using a threshold.

Lastly, I’ve heard that reading bits using digitalRead does a whole bunch of verifications on the inputs.  Why not just read directly from the Atmega328p register and do all the bits at once.  The code at the beginning of “loop” does the necessary bit manipulations to get this to happen using a single read from the PORTC and PORTB registers.  I haven’t converted the output bits to using direct register writes, but they happen so much less frequently, that it won’t make much difference.

So immediately after burning the code, I noticed that the LED which I use to time the loop is much less blinky.  (Before, I could notice a very faint flicker in the LED, but now it seems to be continuously lit dimly.)  That is a good sign.  The real test is when I throw it on the logic analyzer.  So now I get the main loop taking between 23.5 us and 29 us.  That is a significant improvement from the original 488 us.  (1/20th of the time).

So the pulses normally take 4 us, then every once in a while they take 11 us.  This probably means that the interrupt takes about 6.5 us or 7 us.  That also matches up nicely to the 6.5 us difference that I see in the length of time it takes to run the main loop.  It also says that I should take the time to stop using digitalWrite and write directly to the register.

Here is a link to the bare metal version of the code.

11/14/2018 – PSOC vs Arduino nano – Round 1

Finally…today I got my nano boards.  I ordered them on 9/30/2018, and, well 45 days later, I received them.  The main issue that I have is that the seller told me a week after I ordered them that I should expect them within a week.  Seems what he really meant to say is that it would take about another month or more to get them.  Water under the bridge as they say.  (You also get what you pay for, and 5 nanos for only $10.53 isn’t that bad I guess).

So, I soldered the headers on one of the boards, uploaded the program to the board (it uses the old bootloader), and suddenly the little test loop program that could fire solenoids was running.  It was relatively painless except for the fact that the Arduino IDE doesn’t like to program it twice in a row without closing out the program and restarting it.  (It seems to default to looking for the elf file in a location that doesn’t exist).

So I’ve set up the program to act like there are two solenoid wings attached to the board.  A solenoid wing has four inputs, and four MOSFETs for driving a solenoid, so in total there are eight solenoids being polled, and it is controlling eight outputs turning the MOSFETs on and off.  There is enough logic to drive the solenoids for certain period of time for an initial kick and to PWM a single coil flipper.  There is no serial controlling logic to allow a host computer to kick a solenoid, or even read the current state of the inputs from a “rules” computer.  That will come a little later.

At the end of the main loop, I pulse Pin13 (the one attached to the LED) high and then back low to give a quick pulse.  This is a really easy way to do timings to figure out how long it takes to execute the main loop.  After throwing the logic analyzer on Pin13, I got an initial measurement of 488 us.  Hmmm, that is pretty darn slow, especially since I am not doing a lot of the other processing that is necessary to allow a pinball system to work.  Let’s say that the initial pulse is set to 32 ms.  32 ms +/- 488 us has an error of over 1.5%.  I have not tried to “optimize” the program in any way.  There are many websites, that talk about ditching the Arduino framework and bare metal programming the processor.  That will be my next step.  It is an interesting first marker in the sand.

Let’s go over to the PSOC and see how fast it runs its main loop.  I instrumented it up so that bit 4.3 is gives a rising pulse each time the main loop is completed.  (Note:  These timings are with no serial traffic occurring, so there is almost no jitter in the signal).  The PSOC board is configured with the following 4 wings:  neo-pixel, input, solenoid and incandescent.  (The incandescent wing was chosen, because even when idling, it defaults to blinking the incandescent lights so in white wood mode, the lamps can easily be tested)  With that configuration, the main loop occurs every 84.5 us to 86 us, or about 11,500 per second.  The loop rate seems sufficiently high that the error should not affect the power of pop bumpers.  SS3 uses 32 ms for the pop bumpers/slingshots, so 32 ms +/- 86 us or an error of .3%.

As I get more time in the next couple of weeks, I’ll try to bare metal program the Arduino to see if I can get the rate much higher.  Here is a link to the Arduino nano program.

11/3/2018 – Using incandescent wing as HiVolt inputs

Every once in a while, somebody starts emailing me about a technical question and after a couple of emails, a solution is created, and all the information disappears into the ether.  Maybe others could use this information, but I rarely take the time to put the solution out for others to benefit.  Well, since I’m still waiting for my Arduino nano boards, I will take this time to bring up one of those small projects.

A couple months ago a user wrote the following:  “Because of the half scale of my pin, the ball is too light to press a mechanical switch, I use in some instances IR detectors and induction (proximity sensor) switches but induction sensors are 12v (6v to 36v to be precise but I have a 12v psu already).  Is there a simple way to connect it to the processor board without damaging it?”  Originally he proposed using an opto-coupler board as a solution (approx $6.68 for 4 bits + $3 shipping), but suddenly my inherent cheapness caught the better of me.  Could I do it for a much lower cost?

So the basic issue is a 12v input needs to be attached to a processor pin which can only support up to 5v (in the case of the PSOC).  While using an opto-couple board would work wonderfully well, it is just a little too expensive in my opinion for what is needed.  Why not use an incandescent board flipped 180 degrees from how it is normally used?

Could a BS170 transistor make it work?  If the signal goes between 0 and 12V when the IR signal is blocked or not blocked, you could use that as the gate of the transistor.  The drain of the transistor is hooked to the processor input, and the source of the transistor is hooked to the ground of the processor.

Here is a link to the BS170 datasheet:  https://www.onsemi.com/pub/Collateral/BS170-D.PDF

The gate threshold in the On characteristics section of the datasheet says, anything on the gate < 0.8V (the minimum), the transistor is off, so the processor will read high since the input bits are pulled up to 5V internally.  If the gate voltage rises above 3V (the maximum), the transistor will turn on, and the input will be pulled to ground.  BS170 are about 12 cents each if you buy about 100.

The only issue that I can think of off the top of my head is that the sense of the input will be backwards.  Normally a closed switch is read as low.  If it is like an IR sensor, when the sensor is blocked it actually gives low voltage, so that would read high at the processor.  If the IR sensor is not blocked, it gives a high voltage so it would read low at the processor.  If using MPF (Mission Pinball Framework), the input bit can be configured as active high, so it should not be a problem.

On the incandescent wing, the connections that normally attach to the processor go to the gate signals of the BS170s and are attached to the output of the inductor sensor.   Attach the inputs from the processors to what is normally attached to the incandescent bulbs.  (Don’t populate the pullups).  At that point you have 8 of the circuits in a nice little package without having to proto board anything. The gate of a BS170 can handle up to 20V, so your induction sensor at 0-12V should be fine. The BS170 acts as a voltage translator, so that high voltage never touches the processor pins. You will have to tie the logic ground from the processor and the ground of the voltage source for the induction sensor together.

The induction sensor turns on and off the transistor. (turns on and off the switch). The source of the transistor is connected to logic ground. The drain of the transistor is connected to input of the processor. When the inductor sensor sends a high voltage, the transistor or switch closes, so the input is grounded. When the inductor sensor sends a low voltage, the transistor or switch opens, and so the input is floating, but there is an internal pullup resistor in the processor so it reads a high voltage.

Here is an attached picture that he sent me to be a little more clear:

In other news, I moved the firmware from 0.3.0.1 to 1.0.0.0 after re-running all the regression tests and adding a new test for the CAN_CANCEL.  That feature was added a couple weeks back so that the initial kick of a solenoid could be canceled so things like tap passing is supported by the OPP hardware.  (Thanks to Jan for adding the necessary code in MPF to make that happen on their side).

 

10/26/2018 – Waiting, waiting, last call for changes

I kept thinking to myself my next post is going to be after I receive my Arduinos and I can write about my testing.  Unfortunately, when the manufacturer says I should receive them in about a week, it seems like we are nearing a month.  Sigh.  I guess I get what I pay for.  (Of course, I did get five Arduino Nano’s for $10.83.  The only thing that annoys me is that I paid $1.50 of that to get the boards faster, but that doesn’t seem to be working out for me.)

So while I’ve been waiting around for Arduinos, I had a visitor.  Yes, the famous Joe and his wife showed up at my doorstep one Saturday morning.  Joe of course lives in Chicago, and I live in Massachusetts, so it was a little unexpected.  I have been working with Joe for approximately six years at this point.  Joe has done all of the artwork and callouts on the SharpeShooter III pinball machine.  Joe also did the artwork and came up with many of the rules for the Van Halen pinball machine.  Joe owns the Dolly Parton to Van Halen machine.  Unfortunately Joe had never been able to play SharpeShooter III.

Joe and his wife happened to be traveling to Maine for week long vacation to see “more lighthouses than you can shake a stick at,” as he says.  His wife was also interested in driving to the eastern most point in the United States.  It is tough to fault anyone for that since my wife and I have driven to both the western most contiguous point that you can drive to in the United States (it is in Alaska), and the southern most point in the United states (it is at Key West) on the same trip.

My daughter shot some “incognito” photos of Joe to prove that he was able to visit OPP world headquarters.  Joe finally got to play SharpeShooter III.  He mentioned how freaky it is hearing his callouts on a machine that he has never played.

So as we were talking pinball, I showed him around the manufacturing floor at OPP, and he took a gander at the playfield blanks that I was going to be using to start the white wood for Disaster.  One quick look and he said, “Is that 3/4 inch plywood?”  I responded with, “Yes, isn’t that what it is supposed to be.”  He responded it should be 1/2″ plywood, and the more appropriate reason is that all of the pinball parts are meant to be mounted on a 1/2″ thick piece of plywood.  Sigh.  I guess it is back to Home Depot to get another sheet.

Meanwhile…an MPF user requested that a new configuration bit be added so that the initial kick for a flipper could be cancelled.  After discussing it with Jan and verifying what was needed, I added a new configuration bit for solenoid configuration.  Version 0.3.0.1 of the firmware supports this, and Jan added the necessary code for MPF to support it.  It was a fun little change that could be done in a weekend.  This makes it so the ball can be tap passed between the flippers.

So I’m throwing out a last call for OPP firmware changes before I rename the software to rev 1.0.0.0.  There have not been many request for updates to the firmware lately.  (The last two include the cancel of the initial pulse for flippers, and a delayed initial pulse to support solenoids for kickbacks.)  If anybody knows of any other features that are needed in the firmware, I’ll try to make the necessary updates.  If not, I will rerun all the regression tests to verify all the functions and rename the firmware to 1.0.0.0 and be done with it for right now.

10/11/2018 – Arduino and OPP

Well, it is going towards Fall in this neck of the woods, so I started getting a little bit bored.  There are those people who keep saying Arduino, Arduino, Arduino, and I keep saying, “I don’t care.”  A friend pointed out to me that you could purchase Arduino Nano boards for about $2 each.  That might be compelling enough for me to take a look at.  A couple weeks back, I found a seller on Aliexpress and sent my $10 for four Nano boards.  (I paid the extra for a little faster shipping).

While the boards haven’t arrived yet, I started downloading tools and getting my first code ready for the boards when they arrive.  The tools were a breeze to download and install.  I have to give them, it was exceedingly pleasant to deal with the tools.  The tools are pretty basic compared to the tools that are provided by Cypress, but that makes them easier to use versus the more powerful Cypress tools.

Before ordering the boards, I knew that I wanted to use 16 digital I/O from the Nano board.  I did a quick search on the documentation and it mentioned that the ADC pins could be configured as digital pins, so I should be fine.  I then proceeded to order the boards.  Unfortunately, when I started digging into how to write the first program, I realized that the ADC6-ADC7 (A6-A7) pins can’t be configured as digital pins.  They are dedicated analog only pins.  Bah!  I can deal with that.  It just means that those have to only be used as inputs.  It is annoying, but it can be coded around.

So for my first test, I simply want to attach two solenoid wing boards to the Nano.  Because of the insane way that the boards were laid out and the location of the ports, I had to be a little creative.  On the PSOC, the 8 pins of a port are sequential along the edge of the board.  On the Nano, the pins of ports are mixed between each other.  I guess port C is as close as it gets, but of course, as mentioned before, the top two bits are analog inputs only, so that really doesn’t help when reading an external port once and iterating through all the bits.

So one of the solenoid wings will be on the “right” side of the Nano, and will use PC0-PC7.  This means PC4-PC7 (pins A4-A7) will be the inputs (I’ll use analog reads and check thresholds), and PC0-PC3 (pins A0-A3) will be configured as digital outputs for the four solenoids.  The second solenoid wing will be on the “left” side of the Nano, and will use PB1-PB4 as digital inputs (pins 13-16), and PD5-PD7, PB0 (pins 9-12) as digital outputs.  Well at least the Arduino abstraction layer makes up for the fact that the hardware pins are not contiguous, and makes the pin numbering contiguous for the software.

So here is the setup for the code:

void setup() 
{
  // put your setup code here, to run once:

  // setup ADC0 to ADC3 as digital outputs and low 
  for (int pin = A0; pin <= A3; pin++)
  {
    digitalWrite(pin, LOW);
    pinMode(pin, OUTPUT);
  }
  
  // setup ADC4 to ADC7 as analog inputs
  for (int pin = A4; pin <= A7; pin++)
  {
    pinMode(pin, INPUT_PULLUP);
  }

  // setup PD5-7, PB0 (pins 9-12) as digital outputs and low 
  for (int pin = 9; pin <= 12; pin++)
  {
    digitalWrite(pin, LOW);
    pinMode(pin, OUTPUT);
  }
  
  // setup PB1 to PB4 (pins 13-16) as digital inputs 
  for (int pin = 13; pin <= 16; pin++)
  {
    digitalWrite(pin, LOW);
    pinMode(pin, OUTPUT);
  }

  // for debugging loop speed set LED pin 13 as output
  pinMode(LED_PIN, OUTPUT);
}

When setting up the outputs, I first write the value LOW before changing the pin to an output.  That should guarantee that I don’t get all the solenoids briefly firing at the beginning during initialization.  I also set up the LED so I can use it to see how fast the operation loop is.

Now for the fun stuff.  Let’s control some solenoids!  So I first set up a few states for a state machine.  The solenoid can either be idle, initial kick, PWM off, PWM on, or wait for switch off.  So for normal solenoids like pop bumpers, the solenoid will be idle, detect the switch has closed, go to the initial kick state, wait for the correct amount of time to elapse, go to the wait for switch off, wait for the switch to open, then go back to the idle state.  That is as simple as it gets.

The other type of solenoid I want to initially support is a flipper with a PWM for the hold.  (I really have to get myself some dual wound flippers).  For a flipper, the solenoid will be idle, detect the switch has closed, go to the initial kick state, wait for the correct amount of time to elapse, go to the PWM off state, wait for the correct amount of time to elapse, go to the PWM on state, wait for the correct amount of time to elapse, go to the PWM off state, while in either of the PWM states, look for the switch to go open and if so go back to the idle state.

PWM can be configured as followed:

PWM = 0, no PWM so act like a pop bumper, kickout hole, kicker, etc.

PWM = 1, duty cycle 12.5% on, 87.5% off

PWM = 2, duty cycle 25% on, 75% off

PWM = 3, duty cycle 37.5% on, 62.5% off

PWM = 4, duty cycle 50% on, 50% off

PWM = 5, duty cycle 62.5% on, 37.5% off

PWM = 6, duty cycle 75% on, 25% off

PWM = 7, duty cycle 100% on, 0% off, used for dual wound flippers

To make reading input pins and writing output pins easier, I made two little lookup tables at the beginning of the code.  That way, I just need to keep track of which solenoid I’m currently processing and can use a simple loop.

const int INP_PIN_MAP[MAX_NUM_SOL] = { A4, A5, A6, A7, 9, 10, 11, 12 };
const int OUT_PIN_MAP[MAX_NUM_SOL] = { A0, A1, A2, A3, 6, 7, 8, 9 };

Since half of the inputs are analog inputs and half are digital inputs, also need to have some constants for thresholds, and when to do an analogRead vs digitalRead.

That’s about all the code does at this point.  I’m looking forward to getting my Nano boards to test it out and do some measurements on how fast the loop is.  To be a real controller, it will need a bunch of serial commands for scoring, kicking solenoids, configuration, etc, but that can wait until I get the first loop started.

The see all of the code, you can look at it in the repository/Arduino/NanoOPP.  Note:  The code has not been tested whatsoever, so this code is simply for your viewing pleasure.

 

6/26/2018 – Hot Rod and OPP…what happened?

Let’s get some history:  Jeremy is a long time supporter of OPP (Open Pinball Project).  He was one of the original backers of the OPP kickstarter.  He runs the Pinball Makers website and created  many of the instructions on how to build and connect together OPP cards.  I had documentation, but Jeremy had much better skills at putting together better diagrams to make populating the boards more easily understood by a person other than me.  Jeremy rocks!, and I thank him for all of the hard work that he has done over the past three or four years.

Jeremy decided to switch his pinball controllers from OPP to PROC (I really have no idea what the PROC acronym means).  I support anyone’s decision to choose the PROC because it is a very mature platform.  I think of OPP providing a service for hackers mostly.  I think of PROC as a full fledged product.  Side note:  I was talking to Gerry a couple years back, and I believe he thought OPP was stealing his customers.  I told him that I believe that OPP would eventually drive customers to his products.  I simply think this is what happened in this case.

So nobody changes their pinball controller without having a heck of a good reason.  A major change like that involves a significant amount of work for rewiring, and a lot of extra cost for buying those more expensive controllers.

I have rethemed and run two different machines using OPP controllers.  They work without issues, and my main complaint is that their boot time is too long.  Why are they different than Jeremy’s machine?  The main difference is that I use the OPP pinball framework versus using MPF (Mission Pinball Framework).  This points to the Achille’s tendon of OPP boards.  The communication to the boards are slower than all other pinball controllers out there.

Let’s give a bit of history on OPP.  It was meant to control late ’80s style pinball machines.  That was the goal.  Not ’90s style with DMDs and all sorts of bells and whistles, but the much simpler ’80s style with incandescent bulbs and such.  This was mostly based on my knowledge of how much skill I had at making a pinball machine.  I think that I looked at both Firepower 2 and Dolly Parton as a realistic goal of what I needed to be able to control.  Firepower has a lamp matrix so I used that as a goal for what OPP boards needed to support and how frequently updates were required to lamps.

After a little research on lamp matrices, each column is strobed for 2 ms, there are 8 columns, so each light bulb is updated every 16 ms.  An insert light that is “on” is really only on for 2 ms out of 16 ms, but because of persistence of vision, it appears to be on all the time.  Since OPP bulbs are on 100% of the time, there was really no reason to update lamps more quickly than once every 16 ms, and even that, doesn’t make much sense because it will look like a bulb is continuously on.

Another typical mode for a bulb is to blink either rapidly or slowly.  From previous projects, the most rapid blinking that is comfortable to me, and doesn’t just look like flickering is about 100 ms.  A slow blink is about once per second.  Since that is needed so frequently in pinball, (i.e. blink an insert rapidly or slowly) I created OPP commands so that the PSOC 4200 could blink at either fast or slow speeds automatically.  That further reduces the need for serial commands and reduces the need for updates to less than every 100 ms for each board.

The OPP framework uses the above types of commands to minimize serial communication.  How it is actually implemented is it uses a 100 ms tick and collects all updates to lamps during that tick.  At the end of 100 ms, the framework gets what the current state of the incandescent bulbs should be, and sends a single command to update them if they are different than the previously saved state.

On MPF, I tried to implement the same sort of controls (without using the blink commands since there is really no equivalent in MPF).  I thought I got pretty close, but after looking through Jeremy’s log files, I can definitely see that I didn’t achieve what I wanted.  (Since I don’t have an MPF machine, most of the coding that I do in MPF is “blind” coding.  I don’t have a good way to test to code, and really rely on others to make sure it is correct.)

MPF seems to use an individual lamp as an object.  (Object oriented wise, that makes a ton of sense).  The OPP pinball framework uses the whole card as an object.  I organized it that way to make it easier to collect updates to all lamps on a whole PSOC 4200 to minimize the number of  serial commands.  The MPF object view makes aggregation more difficult since individual lamps aren’t tied together into a single object.  I tried to get this to happen but evidently I didn’t do a good job.

One last reason to minimize turning lamps on and off quickly is flyback voltage.  Jeremy kept reporting that the incandescent boards were getting really hot from the BS-170 MOSFETs.  That confused me because my MOSFETs didn’t have that issue.  (Van Halen uses BS170s just like his machine).  What I failed to take into account is that the framework was turning the MOSFETs on and off very quickly.  Just like with solenoids, when an incandescent MOSFET is turned off,  the flyback voltage builds up.  With solenoids, an extra diode is placed across the solenoid to bleed off the voltage.  With an incandescent bulb, the current is much lower so it doesn’t build up as much flyback voltage, so the diode isn’t normally necessary.  But…if the incandescent is switched on and off rapidly, it becomes more of an issue, especially because of heat build up at the junction of semiconductor.

So here is an example from Jeremy’s log files, right about when everything goes wrong:

2018-05-23 06:30:44,861 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x00 0x0f 0x5f 0x36 0x23
2018-05-23 06:30:44,866 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0xdc 0x10 0x4e 0x0e 0xd6
2018-05-23 06:30:44,871 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0xdd 0x10 0x4e 0x0e 0xc0
2018-05-23 06:30:44,875 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9d 0x10 0x4e 0x0e 0x5b
2018-05-23 06:30:44,880 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9f 0x10 0x4e 0x0e 0x77
2018-05-23 06:30:44,884 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x10 0x4e 0x0e 0x2f
2018-05-23 06:30:44,889 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x10 0x5e 0x0e 0x78
2018-05-23 06:30:44,893 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x9b 0x00 0x5e 0x0e 0xda
2018-05-23 06:30:44,898 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x00 0x0f 0x5f 0x37 0x24
2018-05-23 06:30:44,906 : DEBUG : OPP : Update incand cmd: 0x23 0x13 0x07 0x00 0x00 0x00 0x3f 0xf2
2018-05-23 06:30:44,910 : DEBUG : OPP : Update incand cmd: 0x20 0x13 0x07 0x84 0x89 0xf8 0x00 0xcd
2018-05-23 06:30:44,915 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x1b 0x00 0x5e 0x0e 0xeb
2018-05-23 06:30:44,919 : DEBUG : OPP : Update incand cmd: 0x21 0x13 0x07 0x1b 0x01 0x5e 0x0e 0x80

So within 58 ms (44.919 (end timestamp) –  44.861(start timestamp)), 13 update incandescent commands are sent.  Jeremy had eight cards in his system, but they were broken into two strings of four cards each.  The first string has two PSOC4200 cards with incandescent wings.  The second string has a three PSOC4200 with incandescent wings.  To completely update all the incandescent bulbs in the system, it should only take 5 commands.  Anything more than 5 commands will not be perceived by the human eye because of persistence of vision.  Again, I believed that I had implemented this in MPF properly, but I guess I did not.  The second reason I believe this is the issue is that it is right when things failed in his machine.

What else went wrong.  Well, Jeremy reported that he was seeing CRC errors, and when I originally looked into the log files, I couldn’t find any CRC errors.  Because of that I kept emailing Jeremy asking for log files that showed the errors (and ignoring the log files that he sent me), while Jeremy was probably thinking to himself that the log files were showing the errors.  Jeremy was just reporting what he had seen before. (a CRC error looks externally the same, but is really a very different beast).  I assumed that I didn’t receive the correct log file, when I should have realized that the issue might have been described incorrectly.  (Unfortunately, it was also at a very busy time in my real job which meant that I couldn’t spend as much time on it as I usually would)

So instead of communication issues, I should have been looking into recovery from a buffer overflow.  Again, with the OPP pinball framework, the machine never gets buffer overflows because it uses a tick to insure that a buffer overflow can’t happen.  Now, knowing that MPF can easily cause buffer overflows, I need to spend more time to make sure that the OPP firmware can recover gracefully from overflows.

The above discussion may be seen as me pointing the finger at somebody else.  Far from it.  MPF provided a ton of features that Jeremy utilized.  MPF provides stock code for running the ball trough.  (That is all individually coded in the OPP pinball framework and isn’t a nice simple object.)  MPF has stock code for running EM style score reels.  (Score reels require a good amount of code to work properly, and it is definitely not something that the OPP framework would support without hand coding it.)  The OPP framework is straight python code, while MPF offers yaml to make scripting rules easier.  That helps people fearful of coding feel more comfortable.  The biggest thing MPF offers is the graphics capabilities.  Look at the Nightmare Before Christmas pinball machine and see what Mark was able to accomplish using the MPF framework.  (The OPP framework only really supports static images.  When switching songs on the jukebox in Van Halen, there is a single image for each “high-lighted” song selection so it is really just switching the background image)

Well there it is.  That is my retrospective on what I think went wrong.