Scanning speed and velocity mapping of Cybrid

xooorx PNOscan II (optical key position sensing) reduces crosstalk by having you black out the sides of the keys near where they would otherwise reflect into adjacent sensors:

That is an option, thanks for mentioning it.

luns I think the difference in peak goes hand-in-hand with the additional delay shown by your filter. The filter doesn't just attenuate the quantization noise but is also cutting into the wanted signal too. You might be able to narrow the difference by making your filters less aggressive and respond faster to the input.

Thanks for analysis @luns -- One thing that I believe in this discussion is getting a bit overlooked is that we do not and should not care about the velocity of the hammer per se. We care about the most accurate velocity estimation at the time right before hitting the virtual strings and the most accurate estimation for that time and both things done in real time.

Who cares if the velocity estimation is jumpy when the hammer is far (and hence its position measurement more noisy)? Nobody should care for that, only the most accurate time and final velocity -- even though velocity estimation from noisy distance measurement is still an interesting intellectual exercise.

gzpiano Nice progress

Thanks!

gzpiano For my DIY hybrid, I’ve never seen cross-talk (when playing or in raw samples).

Have you explicitly looked at it, by comparing isolated strikes with 3-nearby-note-strikes? I so, am surprised to hear you haven't seen it, because I tested the sensors you are using and I did notice that they are responsive to off-axis presence. However I would not surprised to hear that it might be not a big problem.

gzpiano Also, 8Ksps works great for me. Get to 127 MIDI.

Getting 127 is easy. What is (perhaps) hard is getting all possible values from 1 to 127, with no gaps and a reasonable touch. I say perhaps because I'm not there yet, I need at least another weekend, if not more.

luns If you hold the key of interest at let-off, how does the cross-talk look then? With the hammer closer to the sensor, I'd expect it to occlude the view of farther neighbours.

Yes, that is exactly my point with the smaller sensors which are more easily screened by the comparatively larger hammers. Quick sketching indicates that total geometric shadow will happen only when the hammer is very close, probably closer than we need to start caring about for velocity estimation (but I don't know exactly the "how close we care" yet).

RIP Have you explicitly looked at it, by comparing isolated strikes with 3-nearby-note-strikes? I so, am surprised to hear you haven't seen it, because I tested the sensors you are using and I did notice that they are responsive to off-axis presence. However I would not surprised to hear that it might be not a big problem.

@RIP : unless @gzpiano has changed his original design, I think that in his case the sensors are placed really close to the base of the hammer shank, where the angular movement is about 1cm... At that close distance, the off axis effect is reduced (it is ₁ cm to the correct hammer base and 2 cms to the neighboring hammers, at an angle that almost precludes IR artifacts). Sensing the base of the hammer is a compromise, but it has its benefits (others being that you can paint the entire hammer in black and leave only a small unpainted patch right beneath the sensor, and that you can easily measure the entire travel in a linear fashion (albeit the travel is short and spatial resolution decreases), while when you sense the hammer head, the hammer at rest is so far that the measurement is very noisy.

As a contribution to the off-axis discussion, here's a plot extracted from that data I posted previously, comparing a sensor being struck, to the adjacent channel where nothing is happening.

Note, the y axis is logarithmic. So while one channel can see an adjacent one moving, that movement barely rises out of the background noise.

Now, that said, I've got CNY70s, which I suspect are a bit more directional than Del Vento's much lower-profile sensors.

vagfilm I think that in his case the sensors are placed really close to the base of the hammer shank, where the angular movement is about 1cm...

Ah, that's right, I forgot about that. Thanks for refreshing my memory. Given that, I am not surprised at all.

JayKominek Note, the y axis is logarithmic. So while one channel can see an adjacent one moving, that movement barely rises out of the background noise.

Nice!

JayKominek Now, that said, I've got CNY70s, which I suspect are a bit more directional than Del Vento's much lower-profile sensors.

I've not done a scientific analysis on this, and I have not installed the CNY70s in the piano (yet), but in my on-the-desk tests waving objects around the sensors, the various sensors seemed similar on-axis and off-axis. If anything the smaller sensors are slightly easier screened by the hammer getting closeby. Speaking of which, it looks like your green channel gets a quite nice screening too when the hammer is close enough. Two questions about it:

1. have you tried to find out at what distance that screening starts, and if so is that just pure geometric shadow?
2. How have you mounted the CNY70 compared to the hammer? LED/Sensor are left/right to the hammer or front/back? If left/right, is the green channel the one for the note to the side of the sensor or to the side of the LED?

JayKominek The firmware knows

JayKominek The relevant code is all in HAL _ ADC _ ConvCpltCallback. (extra spaces to avoid formatting.)

Thank you (and odd that github did not notify me of that commit, I asked to be notified about everything…)

JayKominek It's gnarled because it is doing this processing on a set of circular buffers.

It's actually pretty clear. I don't get 100% of it, but I (think 😅) I get most of it, and you placed quite nice comments, so thanks. If necessary we can talk about some technical details later (perhaps in github is more appropriate than here).

JayKominek Del Vento Before getting there (please folks, do not start an OT on whether or not that's useful: nice conversation, but we need a separate thread since this is too long already), did you verify that you were able to generate all the 127 regular note-on velocities?

I was going to worry about that a bit more when I've got a more realistic calibration procedure in place.

That sounds good. I was just commenting on your "the CC#88 does not make any difference" 🤣

JayKominek Del Vento This very thread started on the speculation that the time resolution we have may be inadequate to already generate the 127 normal MIDI values.

Inadequate under the restriction that you're just doing timing between two values being passed.

Yes, I am coming to the same conclusion, but I have not that as a definite answer. Have you?

JayKominek I thought about mentioning that in my first post, but came to the conclusion I refuse to even mention the names of VSTs in pubilc piano forums.

You don't want to get in trouble eh? 🤣 🤣 🤣

Anyway, thanks for the detailed answer.

I realize how lucky I am since when I started working on my project, and without discussing it with anybody (because nobody would have been interested and familiar enough at the time) I naively assumed that the sequential scanning of all the sensor check points would be so fast that I shouldn't bother much about resolution 🤣 Well, it turned out that was only barely true and only when using the latest Teensy board.

And honestly, if I was to start such a project after reading a thread like this one or the other ones by @RIP and the other Cybrid-like projects, I would simply give up and I'm usually a very obstinate person who isn't easily deterred, etc. I'm also a software engineer and have masters in Engineering Physics... Yet, with all that background I would still find these overly technical talks like in this discussion to introduce too many variables and possibilities for something to not be OK and I would have eventually given up because I usually give up if I think there's reasonable chance for things to not be like I expect them to be.

Ignorance is bliss because I had an apparently overly simplified ideas in my head, based on (almost wrong) assumptions about how things are going to work and it's the lack of any other people competing with me that I just finished it without too much fuss and luckily it managed to work decently 😲🤣

CyberGene I would simply give up

I see your point. To not discourage people who may come to this now, let me instead add: we are nitpicking to make it perfect and very competitive (if not much better) than similar commercial instruments. Getting the "basics" to work is very easy and in fact that is the "goal 2B" of https://github.com/davidedelvento/Mybrid/tree/main/RaspberryPiPico -- In fact that is doable much easier than your Cybrid! Once could even use all commercial parts ("goal 3") and at a low cost.

CyberGene Ignorance is bliss because I had an apparently overly simplified ideas in my head, based on (almost wrong) assumptions about how things are going to work

Well, this is very true! In fact many breakthrough in science came from young people who did not know better. The old school folks knew some theorem or experimental result that proved the new discovery impossible, so they did not even try. The blissful ignorant young, in these cases, were able to discover some loopholes or other workarounds from the theorem and the breakthrough happened. If there is interest, open a separate thread for that, I'm sure @vagfilm and I can come up with some concrete examples of this, besides Cybrid 🤣

RIP blissful ignorant young,

Yes, I have been the three at some time in the past. At this moment I am only 2/3 of that...

RIP have you tried to find out at what distance that screening starts

I just took a look with the multimeter and my fingers wiggling the hammers around. I'm going to say the peaks in the green signal occurs when the purple hammer is about 9-10mm out from the sensor. Literally a hand-waving measurement.

RIP How have you mounted the CNY70 compared to the hammer? LED/Sensor are left/right to the hammer or front/back?

The LED and phototransistor are in-line with the motion of the hammer, modulo some slight twist I applied to the PCB to make them line up a bit better with the bass hammers. The LED is towards the rear of the action, and the phototransistor closer to the front of the action.

Which is odd, I thought I had designed it the other way around, for some reason. Not that I expect it matters a whole lot.

RIP If necessary we can talk about some technical details later (perhaps in github is more appropriate than here).

By all means, I'm happy to take questions on it via Github, or whatever.

RIP Yes, I am coming to the same conclusion, but I have not that as a definite answer. Have you?

Nope, but haven't yet attempted to convince myself one way or the other. I keep hoping one of the other folks will produce a definitive answer. 😀

I think I'm just not going to let the issue slow down construction. There will be enough other issues to resolve, and the materials cost of any boards I might have to scrap is simply small enough I don't care. (My bigger concern will be the hassle of recovering parts which aren't available given the global IC shortage.)

When I think I've got it ready, I'll just hire some university piano students for pizza and beer to come test it. 😅

CyberGene And honestly, if I was to start such a project after reading a thread like this one or the other ones by @Del Vento and the other Cybrid-like projects, I would simply give up

I would have given up without trying if you hadn't shown it could be done.

The extra effort is because I'm trying to make my design field-repairable, and good enough for manufacturers who currently only make acoustics to copy. Or at least give them some more bargaining power.

If I'd just wanted something for myself, I think I would've stayed much closer to your design, and been done much sooner with much less worry. 😁

I'm trying to sort out how all the various filters described here compare with a simpler approach. However I see a number of options, have we settled on a "best one"? Perhaps the one @JayKominek is utilizing which is

https://github.com/jkominek/piano-conversion/blob/a8b33c9164264ce179180f5ef9f337700e9bbd1d/firmware/adc-board/Core/Src/main.c#L134-L142

#define FIR_LEN 13
// TODO want this in DTCM also
// scipy.signal.savgol_coeffs(13, 2, deriv=1, use='dot', delta=0.01)
// the delta doesn't matter, so it was chosen to bring the values to ~1.0
float filter[FIR_LEN] = { -3.2967032967033107, -2.7472527472527473,
		-2.197802197802188, -1.64835164835163, -1.0989010989010746,
		-0.5494505494505213, 2.9531932455029164e-14, 0.5494505494505779,
		1.098901098901124, 1.6483516483516676, 2.1978021978022086,
		2.7472527472527473, 3.296703296703284 };

and

https://github.com/jkominek/piano-conversion/blob/a8b33c9164264ce179180f5ef9f337700e9bbd1d/firmware/adc-board/Core/Src/main.c#L1035-L1044

static inline __attribute__((always_inline)) float computefilter(int chan, int idxs[])
{
	float v = 0.0;
	for(int i=0; i<FIR_LEN; i++) {
		// TODO check to see what the compiler-emitted assembly looks like.
		// if we're not using the fused multiply and add instruction, fix it.
		v += filter[i] * sensorbuffer[chan][idxs[i]];
	}
	return v;
}

Consider that:

• We need the velocity only at the time of strike (before it is slowed down by whatever the hammer strikes, obviously)
• The hammer is in free flight from let off to the time of strike
• The previous two points imply that the hammer velocity is approximately constant and more accurately slowed down by gravity

My question is: instead of doing this hyperlocal measurement at the very end (with all the noise implications of measuring only few points all nearby), why we could not measure a few points from let off to strike and make the measurement basically averaged along a much longer travel (possibly considering gravity slow down, for which we have analytical formula)?

And by "few points from let off to strike" I really mean a few points (again, to average noise) and not one point alone as I am currently making (if you looked at my code). My gut feeling tells me that this would be computationally simpler (not a big deal on the STM32, but somewhat on a pico, see below) and probably more accurate for the purpose we need.

Thoughts?

vagfilm No idea about implementing Kalman on pico (Python?)

Only now I noticed that I never responded to this. I guess you can use Python on the pico, but I am using C (like Jay is on the STM32). And, to reiterate, the pico is a testing platform which might (and hopefully will) turn out to be decent enough to be viable for those who don't want to order custom PCBs, but it is not our final goal.

RIP My gut feeling tells me that this would be computationally simpler (not a big deal on the STM32, but somewhat on a pico, see below) and probably more accurate for the purpose we need.

The "linear difference" mechanism you're suggest requires deciding beforehand(/algorithmically) where the start and stop points ought to be.

The "compute a velocity constantly when you're close enough to the peak to matter" trades away CPU cycles, which I have plenty of, for the very simple decision procedure. "Is this the peak velocity? Yes, generate note. No, keep waiting for peak velocity."

I've been nearly hopping with excitement ever since I realized how much simpler the logic is when I've got a constant stream of velocities to work with. I wouldn't return to what you're suggesting unless somebody else wrote the code for it, and then wrote automatic calibration procedures, and proved it all worked better.

RIP with all the noise implications of measuring only few points all nearby

That 13-point Savitzky-Golay looks smooth enough on my data that you could convince me it was generated by some purely analytical process, not subject to noise. I've got the processing power to extend that into 20-something points, easily.

RIP have we settled on a "best one"?

I don't see a need to decide on a best one. I (at least) just need to find a "good enough" one (for now). I'm doing engineering, not science. 😃

I consider the Kalman filter interesting as well. I looked at that sometime back in the fall, but my ADC readings were still too rough, and it couldn't overcome the noise. It might be feasible, now. Since it could also produce a running estimate of velocity, the same simple decision procedure would apply, and there'd be a minimum of calibration.

JayKominek the peak velocity? Yes, generate note. No, keep waiting for peak velocity."

You don't take into account any absolute position? How can you detect a silent strike without accounting for position?

vagfilm

So, I wrote a little quickly since I was addressing DelVento who'd been reading the relevant code. The slightly longer version is, I only run the velocity calculation when the position is "high enough". I consider that more of a performance optimization than something necessary for correctness.

(Hand waving, without reviewing my data: ) Quiet/partial/incomplete/whatever strikes produce wildly low note velocities which convert to MIDI notes with velocity <1. For debugging purposes I round those up to 1, but when the code is more finalized, they'd just be dropped, as even with high resolution MIDI, you can't go below 1.0.

But by only running the velocity code when the hammer is high enough, the code wastes less time, and processes fewer events which would just be dropped later.

JayKominek The slightly longer version is, I only run the velocity calculation when the position is "high enough"

Ok. Thanks for clarifying. I think that is a good approach...
If my logic is correct, when the hammer lefts the resting position and does not return to resting position but also does not reach the max position, then you can send a note-on == 1, and some VST's use that as silent strike with the damper lifted and let the string resonate. Position of the hammer (or position of the key) is important for some events.

On the subject of lightly played, near-silent and silent notes: The velocity peak is when the hammer escapes the key into free flight, but it's slowing down all the way from there to the string (if it even reaches the string). This may be a small difference for loud notes but the quieter a note gets the bigger the difference will be. Maybe better to take the velocity from a position calibrated to be just below the string?

One very basic newtonian question (told you before that I am no expert in physics, even basic things...).
Velocity of hammer at impact always equals sound, or momentum/acceleration plays a role? I mean, if the hammer hits the string at velocity N when played from the jack (ie, hammer accelerating) does it cause the same string displacement (loudness) as an hammer at velocity N that is in free flight from rest position because of a staccato hit (and is probably deaccelerating)? Maybe in practice this does not matter (off the jack may never reach the same impact speed as a strong free flight hit)...

vagfilm Velocity of hammer at impact always equals sound, or momentum/acceleration plays a role?

The difference between velocity and momentum is just the hammer mass, which all else being equal, would certainly matter. You can expect a 20g hammer striking at 6m/s to sound louder than a 10g hammer. However, this mass matters in the same way that string tension, string weight, felt damping, hardness and shape etc, all matter. These are all essentially the same from blow to blow on a given string, so all would get lumped into the string model.

vagfilm I mean, if the hammer hits the string at velocity N when played from the jack (ie, hammer accelerating) does it cause the same string displacement (loudness) as an hammer at velocity N that is in free flight from rest position because of a staccato hit (and is probably deaccelerating)?

This is a slightly different discussion, with perhaps some controversy. In the very simplest model, the hammer is in free flight when it hits the strings regardless of how you attack the key, so the only thing the string can see is the final velocity of the hammer. How you accelerate to get to that velocity, whether as an impulse, or an extended press is irrelevant. As long as you're still holding the key depressed after the strike, I think this is an adequate picture.
However, if you're considering different staccato hits, there can be differences in the timing of when the damper returns to the string relative to the hammer hitting the string. The quick strike can let the damper return to the string sooner and harder for a shorter note. I believe that for very short notes, the perception of loudness is also affected by the note length, so it's not quite apples to apples comparing identical strike velocities. However, if you want to compare notes of comparable loudness, you can have a short duration, high velocity key strike, or a longer duration, lower velocity strike, having the same 'loudness', but different tonality.

RIP And by "few points from let off to strike" I really mean a few points (again, to average noise) and not one point alone as I am currently making (if you looked at my code). My gut feeling tells me that this would be computationally simpler (not a big deal on the STM32, but somewhat on a pico, see below) and probably more accurate for the purpose we need.

The 'few points away from strike' boils down to a trade-off. Fewer points = more bandwidth = more noise, but also less latency.

My feeling is Savitzky-Galoy is overkill, but if the drawbacks aren't show-stoppers, it's perfectly reasonable to accept as being good enough and move on to other matters.

The drawbacks are the computational complexity and latency. In what I see of @JayKominek 's implementation, you have to keep 13 past samples around, and a floating point multiply and addition on each of them, for every cycle. The complexity is a moot point if the platform can provide it, but as you point out, can be an issue say, on the pico. The other matter is latency - the 13 point FIR, assuming it's linear phase, is a 6.5 sample group delay. I don't know what sampling rate this is at, and haven't played with his data, but this is probably neither a show-stopper nor immaterial.

With @gzpiano 's data, last I played with it, it looked like two single-pole FIR filters (each just one state, and a shift/add - this can all be fixed-point) is adequate for smoothing out any peakiness in the velocity for his original 10-bit data. The coefficients I last looked at, I think gave a group delay of 5 cycles, so not significantly better than Jay's, but the simple fixed-point calculations can be a win. OTOH, with the same data, a shorter Savitzky-Golay filter may be adequate too, bringing its delay down too, I haven't tried.

Thanks @luns for the reply. It was strictly a naive Newton' physics question, and I was maybe not clear, but I was referring to differences in free flight travel on the same hammer (so, no change in mass).

luns I believe that for very short notes, the perception of loudness is also affected by the note length

As I said above, my question did not relate to this, but you are correct. To an extent, perception of loudness depends on volume contrast, so very short notes are perceived as louder than slightly longer ones.

luns The other matter is latency - the 13 point FIR, assuming it's linear phase, is a 6.5 sample group delay.

By default scipy.signal.savgol_coeffs generates coefficients which evaluate the polynomial at the centre of the window, but you can get it to generate coefficients to evaluate any position you like including the endpoints:

scipy.signal.savgol_coeffs(13, 2, deriv=1, use='dot', delta=0.01)
generates @JayKominek 's original coefficients, FIR computes derivative at 6 sample delay.

scipy.signal.savgol_coeffs(13, 2, deriv=1, use='dot', delta=0.01, pos=12)
generates different coefficients, FIR computes derivative at "12th" position in window, i.e. current sample.

It's not a free lunch though, the filter becomes worse in other ways (noisier and can overshoot rapid changes) if you move right to the end of the window. But tuning it for things like having a slightly longer FIR yet with a slightly shorter delay on the filter is possible.

JayKominek I've been nearly hopping with excitement ever since I realized how much simpler the logic is when I've got a constant stream of velocities to work with. I wouldn't return to what you're suggesting unless somebody else wrote the code for it, and then wrote automatic calibration procedures, and proved it all worked better.

Well, writing the code is easy, but calibration is tedious and not fully automatic as you say. I have a basic idea but it requires to do a couple of hammer strikes on each key. Getting rid of that is an excellent point which I underappreciated until you stated it so clearly in this message! Thanks!

For the sake of discussion and understanding, I still would like to get an answer to my question, which was slightly different (and I'll get to it in a moment).

luns The 'few points away from strike' boils down to a trade-off. Fewer points = more bandwidth = more noise, but also less latency.

What do you mean less latency? (*) Assuming you have enough compute horsepower to do the math (which I think the STM32 does, perhaps not so on the RPiPico), you can do more points, say 50? 100? Again, these are just "a few".

I mean, we are measuring the hammer from let off (and even before) to strike, that is at least 1mm, perhaps 2mm or more. Ballpark that should be 1000 samples, perhaps more, and we are throwing 90% or more away and utilize only the last few dozen(s) to compute the final velocity. Well, the final velocity is the only thing we need because it is what it causes the string to vibrate in an acoustic, so I am fine with that. What I am asking (and this reconnects with what I was saying above to Jay and in my "stirring" post of this morning) is if this is the best way to proceed (considering also measurement noise) or if utilizing earlier samples could provide better estimates of this end velocity -- regardless (for now) of other complexities, e.g. calibration as Jay mentioned in his answer.

In other words: knowing what we know, i.e.:

• the hammer is in free flight, being slowed down only by gravity (and perhaps friction, which we can also model relatively easily)
• frequency, resolution and noise of the ADC (ok, we don't know much of noise characterization, but still)

Is the best estimate of the final velocity the one which throws away the vast majority of the samples? Perhaps the answer is yes, but I don't know and I don't see it being an obvious yes. Or perhaps the answer is "we don't know" or "it's a close call" or something like that. Or even "no, it's not the best, but it's good enough" and hence we still do it that way, because (as Jay said above) it is so much better from a continuously-self-calibration point of view and that trumps everything else.

Thanks guys (even ones who I am not responding directly, I read you all and appreciate the conversation). This is such a nice discussion both from a pure inquiry point of view, as well as the practical implications of building these instruments!

(*) I assumed that we were already doing what @xooorx just wrote as position in position 12th -- I'm gonna reply to that message separately