luns I've not heard of Savitzky-Golay, but if we're to allow things to get more complicated, elements of what I remember of the Kalman filter seem appropriate.
The Kalman is the go-to in my biomedical field for smoothing and predicting fast and dynamic data. But my (limited) experience applies to a single data stream. I don't know how fast it would need to be for acting on 88 keys. And this kind of filtering may be overkill for simple rotational movement...
vagfilm I don't know how fast it would need to be for acting on 88 keys.
The designs we DIY'ers are talking about (or at least the ones I've paid attention to 🤣) have at least one microcontrollers for just each dozen of keys or so, not 88. Perhaps even one microcontroller per just a couple of keys (even though in that case it tends to be really wimpy one). It might still be too slow, and also
vagfilm this kind of filtering may be overkill for simple rotational movement...
I was speaking of Kalman on Matlab (the only experience I have...). No idea about implementing Kalman on pico (Python?). Over and out because I am getting out of my depth here. You guys keep the ball and run...
vagfilm No idea about implementing Kalman on pico (Python?). Over and out because I am getting out of my depth here. You guys keep the ball and run...
I'd expect Kalman to be overkill - I only brought it up as a last resort if the data pushes us into that corner, but I don't think we're there. Stopping short of Kalman, we have different simplifications we could make, which end up looking like the other approaches we'd want to try anyway.
luns For a hammer position which is the second integral of what can look like a spikey acceleration, I would go for some second order lowpass filter, or basically a simple biquad filter.
So I played just a little bit with @gzpiano 's data. I didn't even work out the coefficients I'd want for a biquad and just did a simple single pole IIR filter with time constant of about 16 samples. I did this with both the raw data (green), and it quantized to 8 bits (red), both scaled up by 256 for convenience of quantizing. Doesn't look half bad:
There's some quantization noise visible in the velocity curve, but it doesn't seem half bad, and I'd expect that going to the biquad would clean that up some more.
So 8 bits is alright, though it remains to be seen how well we can tolerate reducing the sample rate.
I'll have to pick up with this at another time, but if anybody else wants to pick up where I am, it's really only two lines of matlab:
f=filter([1/16],[1, -15/16],x);
v=filter([1 -1],1,f);
I'll add one more nuance. Just for kicks, here's the same plot but with simple derivatives of the unfiltered raw data and 8 bit quantized (also unfiltered) as well. The derivatives of the filtered raw data was barely visible before and totally covered now, but the filtered 8-bit quantized result which is really close, is still visible.
From this, I can tell the raw data is 10 bits.
I'll also add that the filter really just boils down to per-sample updates of:
out += (in-out)/16
You can think of the filter as taking the prior output as an estimate of where the current position is, then using the difference between the input and that estimate to update the estimate. We don't completely trust the new input - it has its share of noise - so the new estimate is actually 1 part input, 15 parts previous estimate.
Rather than tracking only the position, we can obviously do the same thing with velocity too and include that when making the estimation. This would be the biquad filter I wrote of and will try next when I have the time.
luns 1 part input, 15 parts previous estimate
Heh, this is exactly where I ended up except I chose 1 part to 31:
Getting this out of 10-bit data (and with very simple filtering) has made me a lot happier about my 12-bit converters.
RIP I think we are just trying to use that information to infer what is necessary for the hammers. Speaking of which, The N1/2/3 have hammer sensors right? So perhaps their service manual can help. I just gave it a cursory glance (and it's a service not a design manual) so I could not find the speed of their hammer sensors. However they look like comparators to me a-la original @CyberGene design.
I'm a little surprised by what I found with both this and the Kawai ATX service manuals.
Yamaha has gray-scale optical sensors for the hammer and key. I would have expected the hammer sensor to be used for velocity, and key sensor for note-off sensing (basically damper on/off). However, the schematic seems to indicate that the hammer sensors go through a comparator with programmable DAC output for setting the comparator threshold, and only the key sensors are fed to ADCs.
So, it appears the velocity measurement is done at the key! This would mean the NU1 loud-note design flaw is fundamentally there, and it not showing its head is only a matter of the grand action falling faster relative to the key than the upright does.
I won't rule out the possibility I misread the schematic, but this was jarring enough I would have checked things over at the time, unless I had my attention taken away by something else.
Not to be outdone, Kawai's ATX, while it does have three shutter type optical sensors, actually performs an XOR of two of them right at the sensor board before sending results upstream. So in effect, the three sensors are being used as two! In principle, the two that are XOR'd could even have been combined into one sensor. I really wonder if this was marketing driven, wanting to claim 'triple sensor' implying it's better than two-sensors, when the argument for a third sensor is more typically a description of three comparison thresholds. For squish-switch DP sensors, each sensor only has one threshold, but this isn't so for optical shutters.
Puzzling.
I would not be surprised if the Yamaha hammer sensor exists only to double check if the hammer was really thrown (and avoid the "loud note" events that occur in the models that only have the key sensors). They have all the logic done and tested for calculating velocity based only on the key sensors, so it would make no business sense to alter everything...
luns However, the schematic seems to indicate that the hammer sensors go through a comparator with programmable DAC output for setting the comparator threshold, and only the key sensors are fed to ADCs.
If they can re-set the thresholds fast enough they could be using one comparator to detect 2 or more levels on the same stroke?
luns Not to be outdone, Kawai's ATX, while it does have three shutter type optical sensors, actually performs an XOR of two of them right at the sensor board before sending results upstream. So in effect, the three sensors are being used as two! In principle, the two that are XOR'd could even have been combined into one sensor.
The XOR lets you watch two sensors with a single input pin: It goes high when the first shutter closes and back to low when the second shutter closes, so the length of the pulse is the time between the shutters closing.
Original sampled position: Po
Po differentiated: Vo = Po[t] - Po[t-1]
Vo filtered: Vf = (Vo[t] + 15 x Vf[t-1]) / 16
Vf filtered: Vff = (vf[t] + 7 x Vff[t-1]) / 8
And finally, just to check/visualise what has been retained vs what has been lost from the original data:
Vff integrated: Pc = Pc[t-1] + Vff[t]
xooorx Original sampled position: Po
Thank you to you and all the others who are contributing to this discussion, lots of interesting facts and experimentations here.
One problem I am having with the forum, is that I find its format not ideal for this. I am wondering if we can create a (single, shared -- or each one their own) github repository. There it will be easier to have the plot, the data, the formulas, the code for data collection and for data analysis, all in the same place and update and compare them as needed, rather than having to scroll back and forth and wondering "who said what and when, and is it still their last update and how does it compare with the latest update this other person did and why" -- which is what I am going through now… On the forum we can still post "oh, I updated my repo, and the conclusion is that X works great and Y is useless", with link to the evidence. We might use the https://pianoclack.com/forum/d/276-build-your-own-hybrid-piano-step-three-the-electronics (which is almost a clean slate) or perhaps making a new one for that purpose.
What do you all think? I plan to post everything on github wrt my own data, as soon as I fix an issue I am having with the hardware, which (knock, knock) should happen next weekend.
Thanks!
RIP One problem I am having with the forum, is that I find its format not ideal for this. I am wondering if we can create a (single, shared -- or each one their own) github repository. There it will be easier to have the plot, the data, the formulas, the code for data collection and for data analysis, all in the same place and update and compare them as needed
When I've got code and data it will appear in a GitHub repo, but these current graphs are still at the "thinking out loud" stage. This is @gzpiano's provisional data, with filtering that improves every time @luns comments on it, but all I've got right now really is a spreadsheet where I can mess around with the ideas and make nice plots of them.
I finally sorted out all of my ADC issues. Here's some data I recorded off of two keys last night, for anyone to play with.
Those values are from a 16 bit ADC, at 16kHz. The only processing was to flip the values to match the convention that larger numbers =~ more current from the CNY70.
I'll likely push some more data in there over time, without ceremony. I haven't even looked at the signal off the key sticks, in large part because it was usable even when I had ADC problems.
There's a tiny bit of python next to it which represents what I'm currently thinking of implementing as a signal-to-impact algorithm for hammers. I expect some parameters will be made tweakable per-key things, of course.
Once I get that converted into C, on the controller, dealing with the implementation details of processing from a circular buffer, etc, I'll be sending out pseudo-velocities from the keys. Should reach that point over the weekend.
And here is my first data raw as it come out of the sensors. The code generating the data is here and some helper functions to load it are here
Download above expires in a week. Not putting on my repo because I am refraining to put multi-MB files without first understanding fully what they are. If it turns out to be good, I'll upload it there (it's 2.2MB, not huge, but once in the repo, it will stay there forever in all working copies).
The picture below has been generated from the data with the helper code mentioned above. The flat areas are in order: hammer at rest, hammer at let_off (then there's an obviously-not-flat hammer strike), hammer at drop/backcheck, repetition from there without (and then with) fully releasing the key.
The "baby-strikes" on the right are cross-talks from nearby keys, which could be made smaller with geometric screening (like the 3D printed one I mentioned in another post). I am not worried about them since I plan to use separate damper sensors, but they would be very important to fully characterize if you plan to use a singe sensor like @gzpiano because otherwise your code could mistakenly consider a next-semitone legato playing as a non-release of the previous note.
RIP The "baby-strikes" on the right are cross-talks from nearby keys, which could be made smaller with geometric screening (like the 3D printed one I mentioned in another post). I am not worried about them since I plan to use separate damper sensors[...]
It's worth doing something about that cross-talk if you can. If you play e.g. a cluster of three semitones then the middle note will have its measurements affected by both adjacent hammers which could affect velocity measurement as well as key release.
Its actually a bit worse than it looks because this is raw ADC data. After correcting for the curve in the CNY 70 response those baby-strikes will look somewhat bigger.
xooorx It's worth doing something about that cross-talk if you can. If you play e.g. a cluster of three semitones then the middle note will have its measurements affected by both adjacent hammers which could affect velocity measurement as well as key release.
Thanks for taking a look at the data. Just to make sure other people reading this thread are in the same page, this is IR light reflected by nearby hammers, picked up by the "wrong" sensor. So it's optical cross-talk (not electronic).
I do plan to do something, with a bit of optical/mechanical screening. For key release this looks very important, but in my final design key release will have separate sensors, so that's a non-issue for me.
The cross-talk you see are in fact adjacent notes, and I plan to do some tests as you describe. I will do that after some software improvements that I need for the purpose.
Its actually a bit worse than it looks because this is raw ADC data. After correcting for the curve in the CNY 70 response those baby-strikes will look somewhat bigger.
I am not sure how the curve you posted applies. First, I am using the EAITRCA6, not the CNY70 (for which I expect the problem being worse, because of the geometry of its sensors), but let's assume an identical curve applies. As far as I understand, those curves are for an "infinitely large" object (i.e. much larger than the sensor itself) directly in front of the sensor, whereas the problem here with cross-talk is about a smaller object (nearby hammer), being "on the side".
Moreover, I am not sure I will want to "correct for the response", as in "measure absolute hammer distance". The hammer distance is simply a proxy for (MIDI) velocity, so I plan to go straight there without the intermediate proxy of the distance. That said, your argument still somewhat applies, if hammer position of nearby notes increase the current in the one under measurement, that will skew velocity measurement.
With my "I am not worried" I meant that it looks to be small enough (without any mechanical screening, have you seen my pictures? link below) that it can be easily mitigated. I did not mean "I will just pretend it's not there" 🤣
For example, some extra optical screening can be installed, or just by the "natural" optical screening that happens when the hammer is close enough may be all that is needed (see close-up pictures). This is a big advantage of the small sensors and one of the reasons why I discarded the CNY70 option: the EAITRCA6 is much smaller than the CNY70 -- same would apply if one selects the QRE1113, which might be easier for surface mount industrial soldering, which is how I plan to get it assembled, but I digress….
Link to the pictures: https://pianoclack.com/forum/d/276-build-your-own-hybrid-piano-step-three-the-electronics/3
I'm doing the Savitzky-Golay filter to compute the derivative of position, and then just looking for a peak when the position is "close enough". Apropos of the CNY70 signal shape discussion, by looking at the derivative at its peak, we know we're always at the same physical distance relative to the sensor (that distance being "impacting").
Then on the way back down, I send noteoff when the key stick passes a fixed position. I don't have note off velocity in that video, but I have implemented that since. I feel like it makes things a bit nicer, but it was hardly an A/B test.
After I got in note-off velocity, I went ahead and implemented the CC#88 high res velocities. I can't tell a difference. 🤪
RIP First, I am using the EAITRCA6, not the CNY70
Ah OK I had missed that detail.
RIP Moreover, I am not sure I will want to "correct for the response", as in "measure absolute hammer distance". The hammer distance is simply a proxy for (MIDI) velocity, so I plan to go straight there without the intermediate proxy of the distance.
Even if the finished firmware goes "straight there" it might be useful during development to have at least an idea of where "there" is relative to actual hammer velocity. OTOH if it goes straight to some reasonable function of the actual hammer velocity and that then gets tweaked with a curve that feels right to play, then I suppose that will work too.
JayKominek I'm doing the Savitzky-Golay filter to compute the derivative of position, and then just looking for a peak when the position is "close enough". Apropos of the CNY70 signal shape discussion, by looking at the derivative at its peak, we know we're always at the same physical distance relative to the sensor (that distance being "impacting").
Then on the way back down, I send noteoff when the key stick passes a fixed position. I don't have note off velocity in that video, but I have implemented that since. I feel like it makes things a bit nicer, but it was hardly an A/B test.
At the moment, are you doing this in firmware or on the computer? Do you care to share the code or do you prefer to polish it a bit first?
JayKominek After I got in note-off velocity, I went ahead and implemented the CC#88 high res velocities. I can't tell a difference.
Before getting there (please folks, do not start an OT on whether or not that's useful: nice conversation, but we need a separate thread since this is too long already), did you verify that you were able to generate all the 127 regular note-on velocities? This very thread started on the speculation that the time resolution we have may be inadequate to already generate the 127 normal MIDI values. If that is correct (and I am not saying it is) then adding higher resolution values is obviously a moot point. Also, what virtual instrument are you using to tell the difference, if there is one? That may be important too.
JayKominek I'm doing the Savitzky-Golay filter to compute the derivative of position
I ran the data you posted through Savitzky-Golay (using your python script) and also through "Vf" and "Vff" from my post above (using a spreadsheet). The following shows savgol(13) and savgol(23) in red, Vf and Vff in blue:
I think the difference in peak height is due to savgol doing it more accurately, but in terms of smoothing they're pretty comparable.