Scanning speed and velocity mapping of Cybrid

Context: I'm trying to understand deeper the frequency requirements of my ADC, per discussion in https://pianoclack.com/forum/d/243-diy-piano-sensors and particularly https://pianoclack.com/forum/d/243-diy-piano-sensors/29 -- and related one in https://forum.modartt.com/viewtopic.php?id=8956

Back-of-the-envelope: at ff piano level, hammers move 3m/s, the let off is about 1mm from strings (sensor), so the hammers move from let off to string (sensor) hit in about 300us. I know you are using shank sensors, hence both the distance and the speed are different, but the time is the same. So we have to measure a time around 300us and translate it to a MIDI velocity of about 120. Now we want the system to be able to produce velocities of 121 or 119 (assuming MIDI v1 and ignoring MIDI v2). So we want around 1% resolution in the MIDI output. If the MIDI output is simply multiplicative or divisive of the measured time, that means we need 1% resolution of the measured time, i.e. 3us, i.e. 300kHz per channel.

Question: you mentioned a while back (I think it was on PW) that instead you needed an intense "mapping" to provide appropriate MIDI velocities from the measured times. Looking at your code a https://github.com/ekumanov/cybrid/blob/master/src/teensy/main/My_Hybrid_Piano_V1.2.ino#L220-L245 I think shows what you were talking about. I think I understand and can reverse engineer it, but I think a discussion is better. So the question is: how are you mapping from time to MIDI velocity, do you have a plot of the curve you are using with a range of parameter that you tried (and accepted or discarded) -- basically anything that is not already evident from the code section I linked and even anything that you'd like to add to explain about that code even if it's evident.

Thanks!

I'll answer very shortly and broadly without even looking at the code since I'm a bit tired today and will later on make it more precise if needed.

As is known, I measure duration it takes for the hammer to pass through the last two sensor points. From that duration one can calculate velocity. The school formula is:

distance = velocity x time

The distance is the escapement length (because you would like to place the second sensor as close as possible to escapement point, in order to have more time for the hammer to reach the last sensor, i.e. the rail), so the formula becomes:

velocity = escapement distance / duration

I remember I found some paper that measured the hammer velocity of a Yamaha Disclavier and the corresponding MIDI velocity and they deduced that:

MIDI velocity = 57.96 + 71.3 * log10(hammer_velocity_ms)

So, it appeared that translation between hammer velocity in m/s and MIDI velocity 1-127 is a logarithmic one.

This meant that for ever hammer strike I have to do a mathematical division, logarithm, multiplication and addition. That's a lot when you seek performance in a tiny controller which is why I decided to create a precalculated table in memory that would store the corresponding MIDI velocity for every duration. This way I'm not calculating anything in runtime, instead I have a duration in microseconds and lookup the value in the c corresponding array index. This is a bit risky because it can potentially take a lot of memory space but I made some measurements which luckily I keep and I'm copy-pasting them:

Measured duration in microseconds:
12187us -> produces MIDI value of 2 when escapement is 2mm
9140us -> produces MIDI value of 2 when escapement is 1.5mm
6094us -> produces MIDI value of 2 when escapement is 1mm

It meant that even if the escapement is long at 2mm, I'd still have a table with a maximum of 12187 elements which would fit in the Teensy RAM 🙂 However things were not that simple, see below...

By changing the factor in the formula above (71.3) you would map the usable velocity region within the proper range of 127 values, and the addition coefficient (57.96) determines the absolute position of the produced values within the range.

Basically I experimented with Garritan CFX until I found what felt good. The values are in GitHub.

Now, as I said things were not that simple 🙂 I noticed that with just using that static formula the higher notes would produce unnaturally high velocities. I gave it some thought and finally concluded that that the higher you go, the less heavy the hammer is (at least in my action) and thus it's easier to throw it with a high velocity. However it's also lighter, so the higher velocity is negated by the lower hammer mass (and shorter strings) to produce balanced tone which is why it turned out I had to compensate on a per key basis. Meaning that I had to create a separate array for every hammer. But there's not enough RAM to store 88 tables as the one above, so instead I created groups. Luckily most of the hammers in the left half of the keyboard seemed to work with the same coefficient and could reuse the same table. If I'm not mistaken I ended up with 12 groups.

CyberGene I remember I found some paper that measured the hammer velocity of a Yamaha Disclavier and the corresponding MIDI velocity and they deduced that:

MIDI velocity = 57.96 + 71.3 * log10(hammer_velocity_ms)

http://mtg.upf.edu/mosart/papers/p35.pdf

CyberGene I'll answer very shortly and broadly without even looking at the code since I'm a bit tired today and will later on make it more precise if needed.

n-player http://mtg.upf.edu/mosart/papers/p35.pdf

Awesome, thanks both.

So

6094us -> produces MIDI value of 2 when escapement is 1mm

1mm in 6.094ms corresponds to 0.16m/s which from that logarithmic formula corresponds to MIDI velocity 2.

Inverting it for hammer velocity at MIDI velocity 120 shows about 7.4m/s which is consistent with Fig.3 of the paper (granted, it's the same equation 🤣). So in 1ms the hammer would travel 7.4mm, or in other words it would take 135us for a 1mm escapement.

Doing the same thing for MIDI velocity 121 shows 7.7m/s or 131us for a 1mm escapement.

In conclusion, this preliminary analysis says that I need to have at least a 5us resolution, or that each channel needs to be sampled at 200kHz. Not as bad as if it were linear, rather than logarithmic (in such a case I'd need 1% accuracy on that 135us, or 1us or 1MHz). Not impossible, but a bit of a stretch compared to earlier analysis, even Jay was planning for an order of magnitude less, as you can read in his document.

I need to double check all of this, run a few more tests and decide how to proceed.

Seems like what I copy/pasted was not measurements but just reverse calculations from the Yamaha formula. But that doesn’t matter anyway. It was just a rough estimate about the desired mapping array size.

Hmm. This is certainly problematic.

That's a challenging sample rate to process 88 to 176 channels worth of. Even if you stick a microcontroller on every key, or every few keys, getting all of the data into one place in a low latency fashion becomes a bit of a trick. With 30 seconds of thought, I could probably do it with two channels per RP2040, a CPLD every (splitballing) 8-16 channels, and a couple of CPLDs per microcontroller.

I considered having my analog stage perform analog differentiation of the signal, to produce something velocity-esque, but abandoned it for a few reasons:

1. I wouldn't be able to directly see the position anymore, which would be troublesome.
2. There are more (metaphorical) knobs to turn with the design of a differentiator.
3. Analog differentiators aren't even close to ideal; it (probably?) would need to be tuned to respond to the kind of signal we're interested in.
4. Something else. My breadboarded one wasn't working well perhaps? I think I was testing that with lame power supplies on the IR LED, instead of my swanky ultra-stable power supply. That wouldn't have helped.

I tentatively think I could make it work… maybe. Considering the CNY70 and friends aren't linear in distance, do we even think something like the derivative of their signal would work?

(Also: hello, I am alive.)

Jay: welcome...
About the ongoing discussion: is it really necessary to have a formula for velocity, and assume some kind of regularity? Can't it be simply handled as velocity mapping? Getting a minimum and maximum (should not change that much from neighboring keys - CG got it into 12 groups...) and setting a mapping curve that should be very similar to all keys?

vagfilm I'm not sure what distinction you're trying to make and/or worrying about. A mapping is just a piecewise function. Hand-wavingly, I view "Mapping vs arithmetic function" as a time/space tradeoff in the software, which seems to be a rather low-level issue for these discussions.

. . .

JayKominek I considered having my analog stage perform analog differentiation of the signal, to produce something velocity-esque, but abandoned it for a few reasons:

I agree doing this sort of things in an analog stage is tricky. Not that it's impossible, but I would not embark myself into it. On the other hand, in digital, once you have enough resolution, you are just a few lines of code away from implementing a formula of your choice, a recompile away from changing parameters, and not that difficult to do the mapping if memory is more available and faster than CPU time. As such, I think doing the ADC as close as possible to the sensor, and with the fastest reasonable frequency possible (my eyeballing says that number of bits does not matter, provided is 8 or more effective).

JayKominek That's a challenging sample rate to process 88 to 176 channels worth of.

Just to be sure, so far is only back of the envelope math. It might be needed or it might be overkill. One guy in https://forum.modartt.com/viewtopic.php?id=8956 claims that at least one Kawai (and if true I suspect many of them, I doubt they use different design for different instruments) uses a 90us scanning time and that it is more than sufficient to provide the resolution even at high speed. Also, the sensors which have a time response in their datasheet mention something about 20us delay. That might be irrelevant, since it might simply be a 20us delay of the signal going up or down compared to the object being moved, and so not a loss in resolution (as a time shift, 20us is irrelevant). I haven't tried enough the sensors into the action yet (so far I'm sampling with the RP2040, which can do 500kHz total, or 250kHz per channel if used in two channels, at 9 bit effective), to prove or disprove any or all of this.

Back to your

JayKominek getting all of the data into one place in a low latency fashion becomes a bit of a trick

For calibrations and other stuff, all the sampled data needs to go out, and that's a problem as you say. However once one has settled on a calibration, each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no? The crudest thing could be the controller in charge of each ADC just runs (locally) a noise reduction filter like this then counts the time from when a threshold (let off) is passed until another threshold (note hit) is reached (resetting the counting during or after the hammer movement as appropriate), and convert that time into a MIDI velocity by formula or lookup table, spits out that MIDI velocity with the (fixed) note number for the main controller to pass it via MIDI to the VST. Not much data to send out.

As a proof of concept, I intend to do the latter with three RPi Pico (RP2040) each monitoring three hammers for both note on and note off (in this test I'd use the hammers themselves for note off)

JayKominek (Also: hello, I am alive.)

Hello! Nice to have you back on this project.

RIP For calibrations and other stuff, all the sampled data needs to go out, and that's a problem as you say.

I was assuming that during calibration, the software managing the process would put the hardware into a specialized mode (or modes) where the raw data from a very small number of sensors is streamed out at its full rate, and the rest of the sensors are completely ignored.

So the software would decide to calibrate C4, then command the relevant processor to stream all the data for the hammer and key stick sensors while having you do whatever (pp to ff), Then it'd stop streaming from C4's sensors, switch to C#4, and then repeat the process.

However once one has settled on a calibration, each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no?

Yes. The challenge is in consolidating so many different streams of data (even if they're mostly quiet!) into a single one. For instance, in my design, the ADC boards watch the 22 channels, and spit the resulting MIDI velocities or moral equivalents out their RS485 link to the main board. The main board then has 8 incoming RS485 links.

If you've got 44, 88 or 176 incoming data lines, with velocities on them, you've got to have some way to get all of that combined. When I was originally talking about putting a microcontroller on every single sensor, I was envisioning a CAN bus between all of them, with one device picking up the messages. But it turned out you'd actually need the microcontroller and another CAN transceiver part, etc etc complete pain.

The RP2040's specialized hardware might allow it to pass messages through a chain of them, at high speeds. I'd need to review how it works.

Not sure that's clear. Probably a diagram would help.

The crudest thing could be the controller in charge of each ADC just runs (locally) a noise reduction filter like this then counts the time from when a threshold (let off) is passed until another threshold (note hit) is reached (resetting the counting during or after the hammer movement as appropriate), and convert that time into a MIDI velocity by formula or lookup table, spits out that MIDI velocity with the (fixed) note number for the main controller to pass it via MIDI to the VST.

So, two thoughts.

1. I haven't thought very hard about the algorithm at that link, but I'm not necessary concerned about the responsiveness of the filtering algorithm. We've got some room for latency, I think. Rather, when working on the filtering I'm concerned about distorting the signal. If the phase response makes a big move in the middle of the frequency band that our hammer strikes is in, then… that seems bad.
2. I think I'm still holding out some hope that per the discussion at https://github.com/jkominek/piano-conversion/discussions/31 the fact that the ADC has some spatial resolution might save us from needing all of the temporal resolution you're suggesting we need. But, would have to think about it.

Not much data to send out.

Indeed.

RIP each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no?

As far as I understand, that is correct. Midi out signals are not timestamped by the sender and carry time=0. So, if you connect 88 midi out signals into an enormous usb-hub, it would work... You just need to guarantee that all midi streams have the same latency to the receiver.

RIP One guy in https://forum.modartt.com/viewtopic.php?id=8956 claims that at least one Kawai (and if true I suspect many of them, I doubt they use different design for different instruments) uses a 90us scanning time and that it is more than sufficient to provide the resolution even at high speed.

This sounds the same as the VPC1 scanning (details here) but that document says the switches are basically detecting key travel between around 5mm and 10mm depression. Measuring a real piano hammer in its last mm or so of free flight could require 10 or 20 times that rate for the same resolution.

xooorx This sounds the same as the VPC1 scanning

Am I missing something? VPC1 uses mechanical sensors that act as on/off switches, one seemingly activated at 5mm, another at 10mm, and probably another one at the 1-3mm range... Velocity is calculated at the time between switches.

xooorx This sounds the same as the VPC1 scanning (details here)

Thanks for sharing!

xooorx Measuring a real piano hammer in its last mm or so of free flight

Hammers travel about 5x the speed of keys doing 5x the distance. If we measure only free flight we are talking about 1/5 of the distance (I think this is an important design choice compared to non-hybrid actions), hence your back of the envelope

xooorx could require 10 or 20 times that rate for the same resolution.

seems correct.

vagfilm Am I missing something? VPC1 uses mechanical sensors that act as on/off switches,

I think we are just trying to use that information to infer what is necessary for the hammers. Speaking of which, The N1/2/3 have hammer sensors right? So perhaps their service manual can help. I just gave it a cursory glance (and it's a service not a design manual) so I could not find the speed of their hammer sensors. However they look like comparators to me a-la original @CyberGene design.

vagfilm Am I missing something?

I think you could (?) be missing that "this" in my post refers to the information @RIP linked to about Kawai keyboard scanning, which I'm confirming with a more detailed link. I'm not saying it matches his own scanning requirements. (I'm actually saying it doesn't).

JayKominek I think I'm still holding out some hope that per the discussion at https://github.com/jkominek/piano-conversion/discussions/31 the fact that the ADC has some spatial resolution might save us from needing all of the temporal resolution you're suggesting we need. But, would have to think about it.

Yes, I think there is some hope, but thinking about it more explicitly is better, hence all the "noise" that I started on this topic. If I find time to stop monitoring the forum and do some soldering (and solve a couple of RPI Pico sw issue), I may have a 9-note MIDI-complete prototype (*) to install in my action in a couple of days. And that could provide some hard data to discuss.

(*): with the 'CA6 sensors. I could order more QRE1113 (I have only one) and/or CNY70 (I have only three) if needed

In case it is helpful, I put some data here along with short Matlab code to demo/explain.

https://github.com/gzweigle/DIY-Grand-Digital-Piano

And, .. nice forum.

CyberGene Now, as I said things were not that simple 🙂 I noticed that with just using that static formula the higher notes would produce unnaturally high velocities. I gave it some thought and finally concluded that that the higher you go, the less heavy the hammer is (at least in my action) and thus it's easier to throw it with a high velocity. However it's also lighter, so the higher velocity is negated by the lower hammer mass (and shorter strings) to produce balanced tone which is why it turned out I had to compensate on a per key basis

That's a flaw in the VST intended to be used with weightless identical keys over the entire range. You shouldn't have to compensate for that in the controller.

Deleted Yes, you're absolutely right. It's just how Garritan CFX is made because that's what I used for my calibrations. They have mapped velocity and timbre curve of its response to not account for the heightened velocities of the smaller treble hammers as on real pianos. It seems all virtual pianos are programmed that way though.