Scanning speed and velocity mapping of Cybrid

JayKominek I considered having my analog stage perform analog differentiation of the signal, to produce something velocity-esque, but abandoned it for a few reasons:

I agree doing this sort of things in an analog stage is tricky. Not that it's impossible, but I would not embark myself into it. On the other hand, in digital, once you have enough resolution, you are just a few lines of code away from implementing a formula of your choice, a recompile away from changing parameters, and not that difficult to do the mapping if memory is more available and faster than CPU time. As such, I think doing the ADC as close as possible to the sensor, and with the fastest reasonable frequency possible (my eyeballing says that number of bits does not matter, provided is 8 or more effective).

JayKominek That's a challenging sample rate to process 88 to 176 channels worth of.

Just to be sure, so far is only back of the envelope math. It might be needed or it might be overkill. One guy in https://forum.modartt.com/viewtopic.php?id=8956 claims that at least one Kawai (and if true I suspect many of them, I doubt they use different design for different instruments) uses a 90us scanning time and that it is more than sufficient to provide the resolution even at high speed. Also, the sensors which have a time response in their datasheet mention something about 20us delay. That might be irrelevant, since it might simply be a 20us delay of the signal going up or down compared to the object being moved, and so not a loss in resolution (as a time shift, 20us is irrelevant). I haven't tried enough the sensors into the action yet (so far I'm sampling with the RP2040, which can do 500kHz total, or 250kHz per channel if used in two channels, at 9 bit effective), to prove or disprove any or all of this.

Back to your

JayKominek getting all of the data into one place in a low latency fashion becomes a bit of a trick

For calibrations and other stuff, all the sampled data needs to go out, and that's a problem as you say. However once one has settled on a calibration, each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no? The crudest thing could be the controller in charge of each ADC just runs (locally) a noise reduction filter like this then counts the time from when a threshold (let off) is passed until another threshold (note hit) is reached (resetting the counting during or after the hammer movement as appropriate), and convert that time into a MIDI velocity by formula or lookup table, spits out that MIDI velocity with the (fixed) note number for the main controller to pass it via MIDI to the VST. Not much data to send out.

As a proof of concept, I intend to do the latter with three RPi Pico (RP2040) each monitoring three hammers for both note on and note off (in this test I'd use the hammers themselves for note off)

JayKominek (Also: hello, I am alive.)

Hello! Nice to have you back on this project.

RIP For calibrations and other stuff, all the sampled data needs to go out, and that's a problem as you say.

I was assuming that during calibration, the software managing the process would put the hardware into a specialized mode (or modes) where the raw data from a very small number of sensors is streamed out at its full rate, and the rest of the sensors are completely ignored.

So the software would decide to calibrate C4, then command the relevant processor to stream all the data for the hammer and key stick sensors while having you do whatever (pp to ff), Then it'd stop streaming from C4's sensors, switch to C#4, and then repeat the process.

However once one has settled on a calibration, each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no?

Yes. The challenge is in consolidating so many different streams of data (even if they're mostly quiet!) into a single one. For instance, in my design, the ADC boards watch the 22 channels, and spit the resulting MIDI velocities or moral equivalents out their RS485 link to the main board. The main board then has 8 incoming RS485 links.

If you've got 44, 88 or 176 incoming data lines, with velocities on them, you've got to have some way to get all of that combined. When I was originally talking about putting a microcontroller on every single sensor, I was envisioning a CAN bus between all of them, with one device picking up the messages. But it turned out you'd actually need the microcontroller and another CAN transceiver part, etc etc complete pain.

The RP2040's specialized hardware might allow it to pass messages through a chain of them, at high speeds. I'd need to review how it works.

Not sure that's clear. Probably a diagram would help.

The crudest thing could be the controller in charge of each ADC just runs (locally) a noise reduction filter like this then counts the time from when a threshold (let off) is passed until another threshold (note hit) is reached (resetting the counting during or after the hammer movement as appropriate), and convert that time into a MIDI velocity by formula or lookup table, spits out that MIDI velocity with the (fixed) note number for the main controller to pass it via MIDI to the VST.

So, two thoughts.

1. I haven't thought very hard about the algorithm at that link, but I'm not necessary concerned about the responsiveness of the filtering algorithm. We've got some room for latency, I think. Rather, when working on the filtering I'm concerned about distorting the signal. If the phase response makes a big move in the middle of the frequency band that our hammer strikes is in, then… that seems bad.
2. I think I'm still holding out some hope that per the discussion at https://github.com/jkominek/piano-conversion/discussions/31 the fact that the ADC has some spatial resolution might save us from needing all of the temporal resolution you're suggesting we need. But, would have to think about it.

Not much data to send out.

Indeed.

RIP each controller can simply spit out its own note-on and note-off with the corresponding velocity, without the need to consolidate all the data in one place no?

As far as I understand, that is correct. Midi out signals are not timestamped by the sender and carry time=0. So, if you connect 88 midi out signals into an enormous usb-hub, it would work... You just need to guarantee that all midi streams have the same latency to the receiver.

RIP One guy in https://forum.modartt.com/viewtopic.php?id=8956 claims that at least one Kawai (and if true I suspect many of them, I doubt they use different design for different instruments) uses a 90us scanning time and that it is more than sufficient to provide the resolution even at high speed.

This sounds the same as the VPC1 scanning (details here) but that document says the switches are basically detecting key travel between around 5mm and 10mm depression. Measuring a real piano hammer in its last mm or so of free flight could require 10 or 20 times that rate for the same resolution.

xooorx This sounds the same as the VPC1 scanning

Am I missing something? VPC1 uses mechanical sensors that act as on/off switches, one seemingly activated at 5mm, another at 10mm, and probably another one at the 1-3mm range... Velocity is calculated at the time between switches.

xooorx This sounds the same as the VPC1 scanning (details here)

Thanks for sharing!

xooorx Measuring a real piano hammer in its last mm or so of free flight

Hammers travel about 5x the speed of keys doing 5x the distance. If we measure only free flight we are talking about 1/5 of the distance (I think this is an important design choice compared to non-hybrid actions), hence your back of the envelope

xooorx could require 10 or 20 times that rate for the same resolution.

seems correct.

vagfilm Am I missing something? VPC1 uses mechanical sensors that act as on/off switches,

I think we are just trying to use that information to infer what is necessary for the hammers. Speaking of which, The N1/2/3 have hammer sensors right? So perhaps their service manual can help. I just gave it a cursory glance (and it's a service not a design manual) so I could not find the speed of their hammer sensors. However they look like comparators to me a-la original @CyberGene design.

vagfilm Am I missing something?

I think you could (?) be missing that "this" in my post refers to the information @RIP linked to about Kawai keyboard scanning, which I'm confirming with a more detailed link. I'm not saying it matches his own scanning requirements. (I'm actually saying it doesn't).

JayKominek I think I'm still holding out some hope that per the discussion at https://github.com/jkominek/piano-conversion/discussions/31 the fact that the ADC has some spatial resolution might save us from needing all of the temporal resolution you're suggesting we need. But, would have to think about it.

Yes, I think there is some hope, but thinking about it more explicitly is better, hence all the "noise" that I started on this topic. If I find time to stop monitoring the forum and do some soldering (and solve a couple of RPI Pico sw issue), I may have a 9-note MIDI-complete prototype (*) to install in my action in a couple of days. And that could provide some hard data to discuss.

(*): with the 'CA6 sensors. I could order more QRE1113 (I have only one) and/or CNY70 (I have only three) if needed

In case it is helpful, I put some data here along with short Matlab code to demo/explain.

https://github.com/gzweigle/DIY-Grand-Digital-Piano

And, .. nice forum.

CyberGene Now, as I said things were not that simple 🙂 I noticed that with just using that static formula the higher notes would produce unnaturally high velocities. I gave it some thought and finally concluded that that the higher you go, the less heavy the hammer is (at least in my action) and thus it's easier to throw it with a high velocity. However it's also lighter, so the higher velocity is negated by the lower hammer mass (and shorter strings) to produce balanced tone which is why it turned out I had to compensate on a per key basis

That's a flaw in the VST intended to be used with weightless identical keys over the entire range. You shouldn't have to compensate for that in the controller.

Deleted Yes, you're absolutely right. It's just how Garritan CFX is made because that's what I used for my calibrations. They have mapped velocity and timbre curve of its response to not account for the heightened velocities of the smaller treble hammers as on real pianos. It seems all virtual pianos are programmed that way though.

CyberGene The VST audience is not expected to actually play the piano (or be able to). So the plugins are tailored to react to MIDI messages in a predictable way. Any compensation is up to DAW MIDI filters.

My old Kawai ES100 didn't actually compensate for the graded hammer weights in the MIDI controller. Instead the built-in samples were somewhat (over)compensated, leaving the remainder of the compensation to the pianist (who is expected to be able to). I wonder if the samples are still the same after they replaced the action in ES110 with a cheaper GHS-tier one with a much shorter pivot, making the effect much less pronounced. Or if the treble is now just dull.

Just learned about Cybrid yesterday and joined in here to chime in. Very impressed with what I've seen so far! I have some thoughts I wanted to share - more than I really have time for ATM, but we'll see how far I get.

First, discussion of hammer speed is something of a proxy, as I don't know of anything aside from the alpha piano that even measures based on the hammer. The "hammer" sensors I'm aware of are at the shank, or rather the flange. These move much slower than the hammer itself, and on the same order as the key, so in a certain sense, numbers for the velocity measurement end up quite similar to doing key-based sensing.

For relating actual velocity to something we see electrically, time to pass a defined distance of course gives us that. For "digital" sensors like the shutters used in the Kawaii ATX and NV, the distance is defined by the spacings of the shutter edges relative to the sensors, and resolution is limited by the rate at which you can sample the sensors. I recall reading, possibly in the ATX service manual, that Kawaii uses a 110kHz sampling rate, but without knowing the actual distance between sensors being measured, that number alone doesn't tell us that much. I do doubt that the measurement is limited to the few mm of hammer position between let-off and hammer strike.

With an analog sensor like the CNY70, we can of course fake the digital sensing approach with two comparators. Now the accuracy isn't limited just by the sampling rate, but it's also affected by the precision to which you can set the comparator thresholds. With the multiturn trim pots, this probably isn't too bad, but with a 8-bit ADC and soft-defined thresholds, the resolution of said thresholds is probably significant.

But using a multi-bit ADC only to compare against a few thresholds is throwing away information the ADC is conveying, and if we're then concerned about limits of resolution of what's left, well, to me that feels like throwing the baby out with the bathwater.

With ADC output available, just taking the difference between subsequent ADC output samples gives you a value for velocity. There are of course resolution limits, but the fact the velocity is continuous and won't change arbitrarily from one sample to the next lets you also filter that result to take advantage of prior samples too for a better estimate. Better yet, as you're watching the velocity from sample to sample, it's possible to detect when there's a sudden change and I would want to use the peak velocity before this change for the measurement.

Of course this is more computation for the controller to do, and the controller talking to the ADC needs to happen with a regular clock that doesn't vary with the controller's load. On the other hand, this should relax the sampling rate requirement, too. Hard to say which side wins.

Even if we wanted to stick to a 'comparator' based approach, we can do better by making use of the ADC output. If one sample is just below your threshold and the next one well past it, that's a different scenario from one sample being far below the threshold, and the next one just crossing it. In a sense, you can interpolate between ADC samples and find where you threshold lies between sampling instants. Again, this should allow relaxing the sampling rate, but at the expense of needing more computation. The interpolation actually requires a division but on the other hand, this only needs to be done twice per key event rather than every sample of every key.

luns The "hammer" sensors I'm aware of are at the shank, or rather the flange. These move much slower than the hammer itself, and on the same order as the key, so in a certain sense, numbers for the velocity measurement end up quite similar to doing key-based sensing.

Hi and welcome 🙂

It doesn’t matter where the sensors are because I’m measuring the duration between two sensor activations and those two points are set based on where the actual hammer tip is, the first point 2mm before stop rail and the second point at stop rail. So, I know that the actual tip of the hammer traveled 2mm for a certain time and that’s the real velocity I’m measuring and using. Sure, the linear velocity of the shank where the optical sensors are, is lower than the tip of the hammer but even if I measured it, it wouldn’t matter because I would be measuring the same duration. I would use a shorter distance which would lead to lower velocity but even in that case it wouldn’t matter since it’s the relative velocity that matters and needs to be mapped into a range of 0-127.

But other than that I agree with you about the resolution and the waste of an ADC if used to simulate what the trimpots and the comparators do in my solution.

luns Just learned about Cybrid yesterday and joined in here to chime in.

Welcome to the clack, or rather I should say welcome to the clique 🤣

Have you seen the whole (still incomplete) series of my posts, haven't you? https://pianoclack.com/forum/d/163-build-your-own-hybrid-piano-step-one-choosing-the-action -- I think somewhere I partially answered your concern/objection regarding sensing hammers vs shanks, which obviously for the ADC approach matters, unlike the comparators approach as CyberGene has just described. I think having the sensors in either place is fine, but I prefer to work with the hammers, so it's easier from an acoustic action regulation and understanding (among other things). If and when the requested speed of the ADC proves to be unfeasible or too expensive, I'll move to another approach. Being DIY, everyone is free to purse the project the way they see fit, obviously.

luns But using a multi-bit ADC only to compare against a few thresholds is throwing away information the ADC is conveying, and if we're then concerned about limits of resolution of what's left, well, to me that feels like throwing the baby out with the bathwater.

luns Even if we wanted to stick to a 'comparator' based approach, we can do better by making use of the ADC output. If one sample is just below your threshold and the next one well past it, that's a different scenario from one sample being far below the threshold, and the next one just crossing it. In a sense, you can interpolate between ADC samples and find where you threshold lies between sampling instants. Again, this should allow relaxing the sampling rate, but at the expense of needing more computation. The interpolation actually requires a division but on the other hand, this only needs to be done twice per key event rather than every sample of every key.

That is exactly why I think for this project the ADC is a better approach than comparators (with potentiometers or with DAC). In fact, I (eventually) plan to do something like what you describe. However, one has to start somewhere, and hence I'm starting with the very crude algorithm which does simply three comparisons (as if it were three physical switches) and ignores everything else. To reiterate just to start somewhere not as the ending point. I hope to have something to share after the weekend (crude, but working, hw and sw prototype)

For my DIY hybrid version, it continuously computes the velocity with an FIR digital filter (in a microprocessor).  See line 56 of Matlab code in linked github page above.  When I play the piano, it sounds great…..although….. only 8 keys assembled…. no la campanella….

gzpiano For my DIY hybrid version, it continuously computes the velocity with an FIR digital filter

Do you mean this line

dx = filter(b,[1],samples) * 2 * 0.0254 * sample_rate / (M+2);

Where is filter defined? Is it intrinsic matlab function?

Edit: Please ignore the dollar signs above. If you are curious they were supposed to make the above line "look like a formula", but they don't work. The dollars gave me gripe when using them as money unit, continue to give me gripe when using them as function / code delimiters. They are displayed normally in the text, but they are untouchable in edit mode: once you mistakenly type them (as I did) you can never delete them (try if you don't believe me)

RIP I fixed it for you but you need to use backticks, not dollar signs.

luns The "hammer" sensors I'm aware of are at the shank, or rather the flange. [...]

luns I do doubt that the measurement is limited to the few mm of hammer position between let-off and hammer strike.

Looking at the ATX service manual...

1) Let-off is set at 4mm (page 10). Possibly slightly wider than a pure acoustic action to accommodate the stop rail? To make the sensing easier?

2) Taking measurements from figure 4 (page 4) suggests the measurement is taken over a hammer movement of about 3mm, although the shutter on the flange is only moving around half this distance due to its location. The figure may well not be to scale but it is clear anyway from the shape of the shutter that the distance measured is very small.

3) From the section on adjusting the sensor height (page 12):

For instances, if the sensor is located at an extremely high position, the position of emitting sound may be higher than the position where the hammer stops by the stop rail. In this state, no electronic sound will be emitted.
On the contrary, if the sensor is located too low, this may cause deterioration of the functionality in detecting consecutive key stroking.

So... "too high" = past the stop rail, "too low" = repetition stops working i.e. below the let-off, so it seems that "just right" is indeed limited to the few mm of hammer position between let-off and hammer strike.

gzpiano , it continuously computes the velocity with an FIR digital filter (in a microprocessor).

FIR filtering... Clever idea. But using unsupervised realtime FIR filtering, are you able to separate a low velocity hit from a low velocity silent note (ie, the hammer is slowly advanced to 1-2mm of the string and does not touch it because of the jeck slipping under the knuckle? And can you compute strikes that have 2 velocities (ie, slow advance until let off point and the sudden acceleration?). My basic knowledge of FIR makes me think it is good for linear movements not complex "multipeak" so to say... Am I wrong? (most probably because I am above my paygrade here 😊). Asking this because FIR may be an interesting approach to realtime analysis of my non-music related professional data signals, but I never attempted to use it, and maybe I should...