Performance of DIY controller prototype

Starting a new thread with a suitable title to attract interested folks. Not @ - mentioning anybody other than you @gzpiano but feel free to

@gzpiano or reference in a step #24 video on youtube, timestamp 1:03 and 2:38 (maybe latter is my playing skills?)

Frankly I can't notice that happening even after repeated watching (and that is perhaps one of the reasons why I hate videos rather than text descriptions of things -- plus with text I can read at the slow of fast speed of my choice, whereas with videos I can slow down or speed up, but it's not the same -- not to mention seeking that in text is immediate whereas in video is a huge PITA -- but I digress, excuse my rant.....)

@gzpiano occasionally drops or delays a note

I trust you that it does happen and I have some ideas. If my understanding is correct, in this architecture the Teensy is connected to a (number of) ADC chips on the SPI bus, correct? What software are you running on the Teensy (or on whatever else you have directly connected to the SPI bus which is producing a fair amount of data)?
Hypotheses (in order of worst-to-best for the purpose of not dropping nor delaying notes): Linux? Linux stripped down to only the essential? Linux RT kernel stripped down to only the essential? RTOS? Just your own code without any OS? How are you dealing with the SPI bus? Polling? Interrupts? Do you have a buffer/queue or are you doing synchronous real-time stuff? If the latter, how? If the buffer/queue, are you monitoring it for overflowing?

So let me make a wild speculation, not knowing any detail about the architecture of your software stack (this would be the easiest scenario to solve): you are running regular Linux, with all of its drivers and other settings and you are just polling the SPI buffer (rather than using interrupts, also because of the rate at which the SPI produces data the interrupts would not be trivial). An interrupt for some other reason comes for the OS, which then "drops" your code and does other stuff. Things in the buffer accumulate and in the best case a note is delayed, whereas in the worst the buffer overflows and a note is dropped.

Other things:

• does this "OS" have a filesystem or is it on RAMDISK?
• Do you run any language that needs garbage collection such as python or matlab? If so, do you have logs that you can use to correlate (or exclude) the dropped/delayed notes happening at the time of gc running?

@gzpiano Plan for debug is stream data about software out Ethernet port while playing look for clues.

If you can make a RAMDISK and flush it to permanent storage later, that would be better. Running the TCP/IP protocol (and to a less degree even just UDP) will at least make the problem worse, if not also muddy the waters.

@gzpiano I wish Teensy second usb was device not host. for performance I'd rather use usb than din MIDI.

Sure that would have better performance, but I don't think it's relevant here. You weren't playing that many notes that fast to get anywhere close to the din MIDI speed limits (which are actually really hard to get close to)

i really enjoy making the videos but agree they have limitations.

No OS. Yes c++. Architecture looks like example Teensy code.

I tried raspberry pi a while back with linux and the issues you mention were apparent then.

Processor is SPI master. using oscilloscope, nothing drops.

Agree about midi.

Because the single board architecture is something possibly someday someone maybe want to try, my goal is open source it on github. The performance architecture seemed pointless to share.

gzpiano i really enjoy making the videos but agree they have limitations.

Just to be clear, I was ranting against youtube (as I did in the past 🤣) not you or anyone else. This being a hobby, you and anyone else does whatever they deem more fun!

gzpiano No OS. Yes c++. Architecture looks like example Teensy code.

Awesome, so you're already ahead of the game, great. But is it based on polling or on interrupts? Does the Teensy have DMA for SPI? Based on I assume each Teensy connects to at least 44 (and perhaps more) SPI busses. But how do you do that? I doubt the Teensy has enough GPIO to do the "select", how do you avoid collisions? In fact my current thought is do do something similar to what you are doing, but to have something like a bunch of RPi pico's (each pico has 2 separate SPI busses). Each pico would control at full throttle 2 ADCs and spitting out (via I2C, probably) only the note-on or note-off (and perhaps a bit of diagnostic), which obviously would go out at a much reduced rate and could easily be parsed in time by a main board. I don't like that the picos are relatively large, however they cost $4 each so add only a couple of buck for each ADC board, which I find acceptable given what they offer. This is somewhat similar to @JayKominek 's initial design in which for each 22 ADC (there were 180 total, counting the pedals) there is a STM32H MCU (in fact in that case the same chip has both the compute and ADC -- but they are still affected by severe part shortage) and then a main board controlling all of the ADC-controlling boards.

If you want to explore the "pico" option, you can do it with a single board (or perhaps two if you want to check the communications out too) which is negligible amount of money. Since this already what I wanted to do, and similar to what I've already done, I'll be happy to share code and ideas (down to the nitty gritty of the code).

If you prefer to keep your current architecture, does the Teensy have a "debug" mode where you can connect another board (perhaps an all-mighty computer) and see what is happening real time? The RPi pico has that and people rave about it (but I have not tried it).

Alternatively, this still applies

RIP If you can make a RAMDISK and flush it to permanent storage later, that would be better

Even though with no OS that can simply be a piece of memory shared by all parts of the code that's running, on a the largest circular buffer you can afford, and with a "stop" option (perhaps the highest note or a pedal) to freeze its content for examining, as soon as you experience the problem -- assuming that the memory can hold the second or so of data which will still continue to flood the system from the moment you notice to the moment to "push" the stop. Without OS it might be complicated to get timestamps which you might need, but still not impossible...

gzpiano Because the single board architecture is something possibly someday someone maybe want to try, my goal is open source it on github. The performance architecture seemed pointless to share.

In the way I see things, posting everything on github is useful. And I really mean everything, I'm even tempted to dump large (MB or larger) chunks of captured data. I don't do the latter only because otherwise the repo becomes quickly too large and unmanageable. In your case, even if I myself would never try what you have done, I can still find plenty of useful information, worthy as gold, for my own use!! For example:

• the thing I'm struggling now (finding replacement parts for those transistors) -- if I had your hardware design, I could use your replacements (assuming you did it recently enough that they are not under shortage too now, but even if they are, that would help me finding yet another replacement by comparing specs)
• the SPI code to communicate with the ADCs -- even if I use a different platform (e.g. the pico), what you are sending to the ADC to configure and read its conversions would apply almost identically - I find the datasheet very dense and a bit cryptic, having an (additional) example would be great
• do this conversation with cognition of facts, rather than speculation

and last but not least (actually first for me!!)

• documenting for myself what I've done, exactly how, and what I've found. I consult as often other people's github repos as I do mine, because I don't remember what I've done (and the computers and OSes come-and-go, whereas github stay -- until MS decides to kill it, at least, which hopefully is not as soon as computers and OSes get replaced in this household)

So I post as much as possible, even if it's in a "crappy" state, and I am not embarrassed if stuff is not "clean and shiny" code or design. Of course, as I said above, you do whatever you find fun and you are motivated to do, you don't have to do what I do 😉

@gzpiano Note that when selecting the sample rates for the piano I built, I used a combination of Nyquist theory plus thinking about latency, and implementation. That is how I landed at 8K.

I guess this is followup on https://pianoclack.com/forum/d/289-scanning-speed-and-velocity-mapping-of-cybrid/ but I prefer posting here because that thread has already becoming too big. Can you please elaborate on your thoughts here? In that conversation we failed to find something that we could all agree and be on the same page, but I have not realized why 🙂

In particular, I wrote

Inverting it for hammer velocity at MIDI velocity 120 shows about 7.4m/s which is consistent with Fig.3 of the paper (granted, it's the same equation 🤣). So in 1ms the hammer would travel 7.4mm, or in other words it would take 135us for a 1mm escapement.
Doing the same thing for MIDI velocity 121 shows 7.7m/s or 131us for a 1mm escapement.
In conclusion, this preliminary analysis says that I need to have at least a 5us resolution, or that each channel needs to be sampled at 200kHz. Not as bad as if it were linear, rather than logarithmic (in such a case I'd need 1% accuracy on that 135us, or 1us or 1MHz). Not impossible, but a bit of a stretch

Can you please show me a similar argument with your ADC demonstrating that 8kS/s is sufficient? I assume you'd be using its higher spatial resolution, which here I'm completely ignoring in the argument above by fixing distance and considering only time resolution

@JayKominek

Looking at the github data, its 2ms? across escapement for 1 m/s max velocity.

Scaling to 7.7 m/s would be ~250us. Which, given assumptions and approximations, validates your 135us estimate.

Nyquist for a frequency domain view of github data signal properties. What can do with data lowpass filtered to half the sample rate.

gzpiano Looking at the github data, its 2ms? across escapement for 1 m/s max velocity.

Well, that is self-referring. Based on http://mtg.upf.edu/mosart/papers/p35.pdf (e.g. Fig 2, 3 the text etc) it seems that the highest desirable velocities of the hammers reach at the very minimum 3.5m/s and hopefully 7m/s

gzpiano Nyquist for a frequency domain view of github data signal properties. What can do with data lowpass filtered to half the sample rate.

+1

(and thanks again for bearing with me with my insistence....)

@"stem_piano" I created a new repository
https://github.com/gzweigle/Piano-Data

I was wondering if you have investigated those oscillations and come to any conclusions. Some advertisement videos of the WNG action claim that wooden actions do oscillate (they have slowmo video demonstrating the effect) and I was wondering if what you are seeing might be real or an electronic effect.

My best guess is the frame that sensors are mounted on moves up/down for an instant, due to force of shank hitting stop.

Okay, so I did another back-of-the-envelope estimation of this issue, this time reversing the role of time and space, and the result is not so bad. @JayKominek you are probably interested in this (if you haven't already done it on your own).

For this discussion I continue to make the assumption of the hammers reaching 7 m/s velocities during fortissimos, and of a worst case (in terms of time/distance travelled) escapement settings at 1mm (typically it should be regulated at 1.5-2mm).

The velocity of 7 m/s is equivalent to a more conveniently thought of 7 mm/ms, which means that the hammer travels the escapement in 1/7 of ms, i.e. 0.14ms
If we are sampling each channel at 8kS/s that means that there is one sample every 0.12ms, so it's possible to have at most 2 samples during escapement travel, but most likely there will be only one. But let me assume we can always get two samples (if we don't we can increase the letoff or increase the sampling frequency modestly, so let's keep that assumption for now and I'll get back to this at the end of this post).

Assuming there are exactly 16 bits of ENOB data and about 40mm of total hammer travel, and assuming the sensors are linear enough and well biased, let me make the back of the envelope estimation of having 65,536 levels in those 40mm ~ 65mm or about 1,000 levels per mm. Some of these assumptions (linearity for sure) are most likely not valid, but I think that 1,000 levels on the last mm can be achieved with some care into powering the LEDs appropriately and biasing the phototransistor for the purpose.

So let me repeat the previous assumptions:

1. we can measure two data points during hammer travel during letoff
2. the distance between these two data points is about 1/7 mm or 0.14 mm
3. each of these two data points is measured with a resolution of about 1/1000 of a mm

Now, if we further assume that resolution is close enough to the accuracy, we can conclude that we can measure this distance with about 1% accuracy.

Finally, I'm sure that the time accuracy between the samples have negligible inaccuracies, so we can conclude that our v=s/t will have a 1% accuracy too. From previous conversations (based on the same PDF linked above), we know that the actual MIDI velocities must be logarithmic in the raw physical velocities. That translates in 2% raw accuracy at high velocities being enough to get 1% accuracy in MIDI. Of course in all this I am neglecting the Savitzky–Golay filtering (or other sophisticated way of computing raw velocity) but I'm pretty sure those will not change these outcome substantially (especially since I'm doing back-of-the-envelope estimate and not full analysis).

So in conclusion, 8kS/s (or marginally higher) can be adequate, provided that the letoff is regulated in such a way to get two "free flight" samples (or equivalently that the pianist does not have a playing style in which he/she changes the speed at the very end of the keypress) -- and provided that all the other assumptions described above are met. So one does not need super-high sampling rates if one has fast enough, and if one has high-resolution enough, sort of what is hinted at https://github.com/jkominek/piano-conversion/discussions/31

Now (and this brings us back to the one vs two samples taken during let off), this require the software to be written by keeping this fact in mind, otherwise there could be missing notes. @gzpiano this might be the problem you are experiencing which is what triggered this thread? This can happen if the software triggers MIDI note-on only when the hammer is detected "close enough" to the sensor, and computes the velocity only in that circumstance (at least that's how I wrote my first version of my software in an effort to keep compute resources low). It is possible that the hammer is never detected that "close enough" because it travels too fast for the given sampling frequency. However, the software can easily consider the velocity too and estimates the next expected position, and realize that it would never see the hammer closer to the sensor because it would bounce before the next sample is detected. Then it can trigger the MIDI note-on with the best estimate of velocity it has, "regardless" of the position of the hammer (well, not completely "regardless", but with a much larger "close enough" and I'm talking sub-ms time here, so it won't be at the wrong time). This is easily distinguished from a apparently similar case in which the hammer is travelling so slow that it never reaches the sensor, but only if velocity is taken into account to decide whether or not triggering a note-on!

The 8 kilo-samples per sec thing and nyquist-related details definitely matters if you're tying to produce a frequency spectrum plot, right? Such as a magnitude versus frequency plot from samples data. I guess - as somebody mentioned - it also matters if trying to estimate a key velocity. Too much time between samples or too little time matters - depending on what the key velocities are. They face similar considerations in using encoders to estimate angular velocity in motors. If the motor turns relatively super slowly - as in takes forever, then that's one extreme. And then when it turns really relatively quickly - then that's another extreme.

Ideally - if it could somehow be all accommodated. Absolute encoder to monitor position. And another algorithm to monitor and detect movement (of the hammer). And this is just one hammer. And we know there's a whole bunch of them.

And when there's movement, a system to estimate whether the movement is relatively slow, or relatively fast - and in which direction. And another system to keep track of which zone (or zones) the hammer is in.

I have never gone down this path before - with measurements of key velocities, and hammer positions/hammer states etc. I definitely see that a heap of excellent work has been done here, and by Yamaha, Kawai etc (which no doubt the work has collectively been a contribution or culmination from lots of people). This is in the hardware and measurements and software control/logic/control departments and more. Outstanding work! Thumbs up.

Forgot to mention the escapement conditions as well. Definitely appreciating the hard yards that had been put into of that.

Math looks good.

Looking at github data, the velocity changes continuously during hammer travel. For example hammer is decelerating under the force of gravity during last moments of free flight.

So where to measure velocity? one picometer before hammer hits the "string" of course! sample rate > 1,000,000,000 per second.

But if I can't do that, then other options (approximations) become possible and my algorithm is not trying to measure the velocity after escapement.

hehe …. that's the main catch. If all the resources and focus can be put into one note or key, then that would be great. But with many keys and lots of processing and relatively high data rate …..

But I must say, got to hand to the people that have gotten the technology to this stsge. Impressive.

I was wondering how fast and reliable the ADC is when dealing with a very steep voltage change that comes from a sensor measuring a very fast hammer? Are there any specs regarding that? Because the ADC relies on charging a capacitor which is a time based thing and when having constant voltage you may increase the sampling rate (hence decrease the single sample duration) but when that’s also linked to a highly variable voltage, maybe the precision drops too?

CyberGene Great question. Here is related discussion about that kind of thing at : LINK

CyberGene I was wondering how fast and reliable the ADC is when dealing with a very steep voltage change that comes from a sensor measuring a very fast hammer? Are there any specs regarding that? Because the ADC relies on charging a capacitor which is a time based thing and when having constant voltage you may increase the sampling rate (hence decrease the single sample duration) but when that’s also linked to a highly variable voltage, maybe the precision drops too?

Sorry I've been quite busy and did not see this before. Certainly @JayKominek can answer more authoritatively, but here my sloppy version.

1. I do not think all ADCs are based on charging a capacitor, but some certainly are
2. The specs that I have seen (for example for the STM32H743 ADCs) explicitly mention the time it takes for the capacitor to charge as a limiting factor. Most of the discussion I saw was about the parasitic resistance of the multiplexer slowing down the charge of the capacitor, and hence having slower sampling rate when using that particular signal route compared to other available options which bypassed that multiplexer. So, yes, I think that the specs from reliable vendors keep this into account.

SouthPark The 8 kilo-samples per sec thing and nyquist-related details definitely matters if you're tying to produce a frequency spectrum plot, right?

Ditto for the time it took me to respond.

I guess that could be a data analysis thing one wants to make but here we are talking about key presses, so we are interested in accurate time-domain things.

SouthPark I guess - as somebody mentioned - it also matters if trying to estimate a key velocity.

That's the whole purpose here: estimate the exact time of keypress (extremely easy with available electronics) and the velocity of the same (not so easy but still feasible), to recreate one of these hybrid pianos, hopefully with even better performance than the commercial ones (and certainly at a more affordable price).

SouthPark Outstanding work! Thumbs up.

Thanks. It takes a village and I'm just the street sweeper, so to speak 😃