Scanning speed and velocity mapping of Cybrid

xooorx It's worth doing something about that cross-talk if you can. If you play e.g. a cluster of three semitones then the middle note will have its measurements affected by both adjacent hammers which could affect velocity measurement as well as key release.

Thanks for taking a look at the data. Just to make sure other people reading this thread are in the same page, this is IR light reflected by nearby hammers, picked up by the "wrong" sensor. So it's optical cross-talk (not electronic).

I do plan to do something, with a bit of optical/mechanical screening. For key release this looks very important, but in my final design key release will have separate sensors, so that's a non-issue for me.

The cross-talk you see are in fact adjacent notes, and I plan to do some tests as you describe. I will do that after some software improvements that I need for the purpose.

Its actually a bit worse than it looks because this is raw ADC data. After correcting for the curve in the CNY 70 response those baby-strikes will look somewhat bigger.

I am not sure how the curve you posted applies. First, I am using the EAITRCA6, not the CNY70 (for which I expect the problem being worse, because of the geometry of its sensors), but let's assume an identical curve applies. As far as I understand, those curves are for an "infinitely large" object (i.e. much larger than the sensor itself) directly in front of the sensor, whereas the problem here with cross-talk is about a smaller object (nearby hammer), being "on the side".
Moreover, I am not sure I will want to "correct for the response", as in "measure absolute hammer distance". The hammer distance is simply a proxy for (MIDI) velocity, so I plan to go straight there without the intermediate proxy of the distance. That said, your argument still somewhat applies, if hammer position of nearby notes increase the current in the one under measurement, that will skew velocity measurement.

With my "I am not worried" I meant that it looks to be small enough (without any mechanical screening, have you seen my pictures? link below) that it can be easily mitigated. I did not mean "I will just pretend it's not there" 🤣

For example, some extra optical screening can be installed, or just by the "natural" optical screening that happens when the hammer is close enough may be all that is needed (see close-up pictures). This is a big advantage of the small sensors and one of the reasons why I discarded the CNY70 option: the EAITRCA6 is much smaller than the CNY70 -- same would apply if one selects the QRE1113, which might be easier for surface mount industrial soldering, which is how I plan to get it assembled, but I digress….

Link to the pictures: https://pianoclack.com/forum/d/276-build-your-own-hybrid-piano-step-three-the-electronics/3

I'm doing the Savitzky-Golay filter to compute the derivative of position, and then just looking for a peak when the position is "close enough". Apropos of the CNY70 signal shape discussion, by looking at the derivative at its peak, we know we're always at the same physical distance relative to the sensor (that distance being "impacting").

Then on the way back down, I send noteoff when the key stick passes a fixed position. I don't have note off velocity in that video, but I have implemented that since. I feel like it makes things a bit nicer, but it was hardly an A/B test.

After I got in note-off velocity, I went ahead and implemented the CC#88 high res velocities. I can't tell a difference. 🤪

RIP First, I am using the EAITRCA6, not the CNY70

Ah OK I had missed that detail.

RIP Moreover, I am not sure I will want to "correct for the response", as in "measure absolute hammer distance". The hammer distance is simply a proxy for (MIDI) velocity, so I plan to go straight there without the intermediate proxy of the distance.

Even if the finished firmware goes "straight there" it might be useful during development to have at least an idea of where "there" is relative to actual hammer velocity. OTOH if it goes straight to some reasonable function of the actual hammer velocity and that then gets tweaked with a curve that feels right to play, then I suppose that will work too.

JayKominek I'm doing the Savitzky-Golay filter to compute the derivative of position, and then just looking for a peak when the position is "close enough". Apropos of the CNY70 signal shape discussion, by looking at the derivative at its peak, we know we're always at the same physical distance relative to the sensor (that distance being "impacting").

Then on the way back down, I send noteoff when the key stick passes a fixed position. I don't have note off velocity in that video, but I have implemented that since. I feel like it makes things a bit nicer, but it was hardly an A/B test.

At the moment, are you doing this in firmware or on the computer? Do you care to share the code or do you prefer to polish it a bit first?

JayKominek After I got in note-off velocity, I went ahead and implemented the CC#88 high res velocities. I can't tell a difference.

Before getting there (please folks, do not start an OT on whether or not that's useful: nice conversation, but we need a separate thread since this is too long already), did you verify that you were able to generate all the 127 regular note-on velocities? This very thread started on the speculation that the time resolution we have may be inadequate to already generate the 127 normal MIDI values. If that is correct (and I am not saying it is) then adding higher resolution values is obviously a moot point. Also, what virtual instrument are you using to tell the difference, if there is one? That may be important too.

JayKominek I'm doing the Savitzky-Golay filter to compute the derivative of position

I ran the data you posted through Savitzky-Golay (using your python script) and also through "Vf" and "Vff" from my post above (using a spreadsheet). The following shows savgol(13) and savgol(23) in red, Vf and Vff in blue:

I think the difference in peak height is due to savgol doing it more accurately, but in terms of smoothing they're pretty comparable.

Sorry for asking this silly question instead of reading the entire discussion (and it's a shame that it's me asking this question but I've been busy with some other stuff and have not been following the progress of this project in details), what are these filters you're discussing? I mean, what are they for? I'd assume you're actually sampling the hammer through ADC and then smoothing the samples in order to produce a curve of the hammer position, is that why you are using filters? If so, then how do you calculate velocity? By measuring the distance of the curve between predefined points?

RIP At the moment, are you doing this in firmware or on the computer? Do you care to share the code or do you prefer to polish it a bit first?

The firmware knows which sensors are hammers and which are dampers, and knows under what circumstances to compute a "pseudovelocity" and send that out. After that, a PC is picking that pseudovelocity up, scaling it to a range appropriate for a hammer or damper, and then running the result through the equation in the Goebl-Bresin paper to get a floating-point MIDI velocity. I "split" the value on the decimal point, and get the base velocity and the high-res portion.

(Long term, everything the PC does will be done by the "main board" in my design, and it'll just emit MIDI via normal routes. I think you know that, but for the benefit of everyone else reading.)

The relevant code is all in HAL _ ADC _ ConvCpltCallback. (extra spaces to avoid formatting.)

https://github.com/jkominek/piano-conversion/blob/a8b33c9164264ce179180f5ef9f337700e9bbd1d/firmware/adc-board/Core/Src/main.c#L1047

It's gnarled because it is doing this processing on a set of circular buffers.

RIP Before getting there (please folks, do not start an OT on whether or not that's useful: nice conversation, but we need a separate thread since this is too long already), did you verify that you were able to generate all the 127 regular note-on velocities?

I was going to worry about that a bit more when I've got a more realistic calibration procedure in place.

I pounded on a key a bunch, and a saw a lot of different values pop out. Did I see 126 and 127? Nope. Did I try to get 126 and 127? Nope. Can my currently configured scaling produce 126 and 127? Possibly not.

RIP This very thread started on the speculation that the time resolution we have may be inadequate to already generate the 127 normal MIDI values.

Inadequate under the restriction that you're just doing timing between two values being passed.

RIP Also, what virtual instrument are you using to tell the difference, if there is one?

I thought about mentioning that in my first post, but came to the conclusion I refuse to even mention the names of VSTs in pubilc piano forums. I'm still using the one I discussed with you previously, and if you've forgotten, feel free to inquire again on my Github project discussion and I'll answer any questions. It might be more suitable to get into the weeds of my system there, anyways.

CyberGene what are these filters you're discussing? I mean, what are they for? I'd assume you're actually sampling the hammer through ADC and then smoothing the samples in order to produce a curve of the hammer position, is that why you are using filters? If so, then how do you calculate velocity?

The Savitzky-Galoy I'm using can both smooth the data, and compute the derivative in a single FIR filter. So that cleans up any small bit of noise in the ADC signal, and gives me a derivative based on multiple close-by data points, rather than having to compute the slope from just two points spread out in time (like the Cybrid effectively did). That should produce a more instantaneous velocity measurement at time of impact, rather than averaging it out over the last 8-10mm of approach.

CyberGene what are these filters you're discussing?

For the curves shown in my most recent post:

Savitzky-Golay calculates a smooth analytical curve through the raw position samples. That's "analytical" as in it knows the equation of the curve, so velocity is simply and directly the first differential of that equation.

Vff numerically differentiates the raw position samples to go straight to a very noisy and nasty velocity measurement, which is then low pass filtered to get a much smoother velocity measurement.

CyberGene If so, then how do you calculate velocity? By measuring the distance of the curve between predefined points?

The curves above are the calculated velocity as the hammer moves. It's finished and done, the graphs are just showing the results.

RIP this is IR light reflected by nearby hammers, picked up by the "wrong" sensor.

Silly and naive question: if you cover the action with the cabinet, do you still have the "baby strikes"? That gets rid of many of the sunlight reflections and could be enough to make the effect irrelevant.

vagfilm Del Vento this is IR light reflected by nearby hammers, picked up by the "wrong" sensor.

Silly and naive question: if you cover the action with the cabinet, do you still have the "baby strikes"? That gets rid of many of the sunlight reflections and could be enough to make the effect irrelevant.

The data was collected at nighttime with minimal (but still comfortable) light in the room.

I have not tried complete obscurity (it's in my todo list already), but I expect to not make any difference for this issue.

I think these baby peaks are caused by IR-LED light (from either the "right" or "wrong" LED) reflected by the "wrong" hammer and collected by the "wrong" phototransistor when playing the next neighbor note (here by "right" I mean assigned to the note which is being played and by "wrong" I mean assigned to a note semitone higher or lower)

EDIT: nothing is silly and naive, every inquire helps increasing the (collective) understanding of the design, thanks for posting

RIP The data was collected at nighttime with minimal (but still comfortable) light in the room.

In that case, only a long pinhole as you propose will get rid of the reflections, but will probably create additional calibration problems because a narrow pinhole will be aimed at different positions of the hammer head as the hammer travels...

I know you really want to use the velocity at the hammer head and not the shank, but if the sensor was aimed at the shank base (in a place where the total travel would be around 10mm) maybe the effect of IR noise would be smaller.
In any case, I would do a test with lights off...

PNOscan II (optical key position sensing) reduces crosstalk by having you black out the sides of the keys near where they would otherwise reflect into adjacent sensors:

So that could be a solution if you don't mind taking a marker pen to the sides of your hammers.

xooorx Original sampled position: Po
Po differentiated: Vo = Po[t] - Po[t-1]
Vo filtered: Vf = (Vo[t] + 15 x Vf[t-1]) / 16
Vf filtered: Vff = (vf[t] + 7 x Vff[t-1]) / 8

This is a particular case of biquad, in this case overdamped. The group delay is 16+8 samples, while high-frequency attenuation is proportional to 16*8=128. I'm not recommending any specific values, but if you went with 12+12 (critically damped) you'd get slightly better high-frequency attenuation of 12*12 =144 for the same delay.

1/12 isn't a great value to work with nor is the other example I'd give of 1/8/sqrt(2), but my gut feeling is you could get away just fine with both layers of filter being 1/8.

This relationship between delay and high-frequency attenuation can be pushed further by going to an underdamped biquad. In particular, I would want to try out something close to a butterworth filter; it can cut the delay by a factor of sqrt(2) (relative to critically damped), and boosts the wanted signal level by as much around the corner frequency.

xooorx I think the difference in peak height is due to savgol doing it more accurately, but in terms of smoothing they're pretty comparable.

I think the difference in peak goes hand-in-hand with the additional delay shown by your filter. The filter doesn't just attenuate the quantization noise but is also cutting into the wanted signal too. You might be able to narrow the difference by making your filters less aggressive and respond faster to the input.

Nice progress @JayKominek and @RIP  I put links to your project on my github page.

For my DIY hybrid, I’ve never seen cross-talk (when playing or in raw samples).  In many lighting conditions. Perhaps that is because of what @vagfilm mentioned.

Also, 8Ksps works great for me. Get to 127 MIDI.

Its configurable 500sps to 32Ksps. 32K is for Mozart, 500 is for Van Halen.

Interesting ideas @xooorx.

CyberGene I mean, what are they for? I'd assume you're actually sampling the hammer through ADC and then smoothing the samples in order to produce a curve of the hammer position

Smoothing == filtering

RIP The "baby-strikes" on the right are cross-talks from nearby keys, which could be made smaller with geometric screening (like the 3D printed one I mentioned in another post).

If you hold the key of interest at let-off, how does the cross-talk look then? With the hammer closer to the sensor, I'd expect it to occlude the view of farther neighbours.

xooorx PNOscan II (optical key position sensing) reduces crosstalk by having you black out the sides of the keys near where they would otherwise reflect into adjacent sensors:

That is an option, thanks for mentioning it.

luns I think the difference in peak goes hand-in-hand with the additional delay shown by your filter. The filter doesn't just attenuate the quantization noise but is also cutting into the wanted signal too. You might be able to narrow the difference by making your filters less aggressive and respond faster to the input.

Thanks for analysis @luns -- One thing that I believe in this discussion is getting a bit overlooked is that we do not and should not care about the velocity of the hammer per se. We care about the most accurate velocity estimation at the time right before hitting the virtual strings and the most accurate estimation for that time and both things done in real time.

Who cares if the velocity estimation is jumpy when the hammer is far (and hence its position measurement more noisy)? Nobody should care for that, only the most accurate time and final velocity -- even though velocity estimation from noisy distance measurement is still an interesting intellectual exercise.

gzpiano Nice progress

Thanks!

gzpiano For my DIY hybrid, I’ve never seen cross-talk (when playing or in raw samples).

Have you explicitly looked at it, by comparing isolated strikes with 3-nearby-note-strikes? I so, am surprised to hear you haven't seen it, because I tested the sensors you are using and I did notice that they are responsive to off-axis presence. However I would not surprised to hear that it might be not a big problem.

gzpiano Also, 8Ksps works great for me. Get to 127 MIDI.

Getting 127 is easy. What is (perhaps) hard is getting all possible values from 1 to 127, with no gaps and a reasonable touch. I say perhaps because I'm not there yet, I need at least another weekend, if not more.

luns If you hold the key of interest at let-off, how does the cross-talk look then? With the hammer closer to the sensor, I'd expect it to occlude the view of farther neighbours.

Yes, that is exactly my point with the smaller sensors which are more easily screened by the comparatively larger hammers. Quick sketching indicates that total geometric shadow will happen only when the hammer is very close, probably closer than we need to start caring about for velocity estimation (but I don't know exactly the "how close we care" yet).

RIP Have you explicitly looked at it, by comparing isolated strikes with 3-nearby-note-strikes? I so, am surprised to hear you haven't seen it, because I tested the sensors you are using and I did notice that they are responsive to off-axis presence. However I would not surprised to hear that it might be not a big problem.

@RIP : unless @gzpiano has changed his original design, I think that in his case the sensors are placed really close to the base of the hammer shank, where the angular movement is about 1cm... At that close distance, the off axis effect is reduced (it is ₁ cm to the correct hammer base and 2 cms to the neighboring hammers, at an angle that almost precludes IR artifacts). Sensing the base of the hammer is a compromise, but it has its benefits (others being that you can paint the entire hammer in black and leave only a small unpainted patch right beneath the sensor, and that you can easily measure the entire travel in a linear fashion (albeit the travel is short and spatial resolution decreases), while when you sense the hammer head, the hammer at rest is so far that the measurement is very noisy.

As a contribution to the off-axis discussion, here's a plot extracted from that data I posted previously, comparing a sensor being struck, to the adjacent channel where nothing is happening.

Note, the y axis is logarithmic. So while one channel can see an adjacent one moving, that movement barely rises out of the background noise.

Now, that said, I've got CNY70s, which I suspect are a bit more directional than Del Vento's much lower-profile sensors.

vagfilm I think that in his case the sensors are placed really close to the base of the hammer shank, where the angular movement is about 1cm...

Ah, that's right, I forgot about that. Thanks for refreshing my memory. Given that, I am not surprised at all.

JayKominek Note, the y axis is logarithmic. So while one channel can see an adjacent one moving, that movement barely rises out of the background noise.

Nice!

JayKominek Now, that said, I've got CNY70s, which I suspect are a bit more directional than Del Vento's much lower-profile sensors.

I've not done a scientific analysis on this, and I have not installed the CNY70s in the piano (yet), but in my on-the-desk tests waving objects around the sensors, the various sensors seemed similar on-axis and off-axis. If anything the smaller sensors are slightly easier screened by the hammer getting closeby. Speaking of which, it looks like your green channel gets a quite nice screening too when the hammer is close enough. Two questions about it:

1. have you tried to find out at what distance that screening starts, and if so is that just pure geometric shadow?
2. How have you mounted the CNY70 compared to the hammer? LED/Sensor are left/right to the hammer or front/back? If left/right, is the green channel the one for the note to the side of the sensor or to the side of the LED?

JayKominek The firmware knows

JayKominek The relevant code is all in HAL _ ADC _ ConvCpltCallback. (extra spaces to avoid formatting.)

Thank you (and odd that github did not notify me of that commit, I asked to be notified about everything…)

JayKominek It's gnarled because it is doing this processing on a set of circular buffers.

It's actually pretty clear. I don't get 100% of it, but I (think 😅) I get most of it, and you placed quite nice comments, so thanks. If necessary we can talk about some technical details later (perhaps in github is more appropriate than here).

JayKominek Del Vento Before getting there (please folks, do not start an OT on whether or not that's useful: nice conversation, but we need a separate thread since this is too long already), did you verify that you were able to generate all the 127 regular note-on velocities?

I was going to worry about that a bit more when I've got a more realistic calibration procedure in place.

That sounds good. I was just commenting on your "the CC#88 does not make any difference" 🤣

JayKominek Del Vento This very thread started on the speculation that the time resolution we have may be inadequate to already generate the 127 normal MIDI values.

Inadequate under the restriction that you're just doing timing between two values being passed.

Yes, I am coming to the same conclusion, but I have not that as a definite answer. Have you?

JayKominek I thought about mentioning that in my first post, but came to the conclusion I refuse to even mention the names of VSTs in pubilc piano forums.

You don't want to get in trouble eh? 🤣 🤣 🤣

Anyway, thanks for the detailed answer.