How to deal with hyperlocality in Savitzky–Golay filtering

RIP · 2022-02-13T15:19:52+00:00

This is a follow up of a conversation we were having in https://pianoclack.com/forum/d/289-scanning-speed-and-velocity-mapping-of-cybrid/103 (and many other posts in that thread). I captured my hammer movements with ADC, and produced velocities with both a comparator approach and a Sav-Gol filter. For comparator approach, I simply mean: timing how long the hammer takes to pass between two arbitrary (but carefully chosen) points. The filter, instead, simply use the savgol coefficients to estimate the velocity at the same place as the second point of the comparator approach: Here are the results Quite different, aren't they? Let's look at the ADC data: Ok, not enough resolution to discern, even if some clues appear. Let's zoom in the first note: And the second: Quite different. So it makes sense to see that difference in overall velocity. Now, with this data, it seems to me that the comparator approach is better (and I hate to say, because I like the Sav-Gol filter better from a theoretical point of view). @JayKominek do you have enough captured data to provide what it looks like with a better ADC capture hardware (even if at lower frequency)? Both the code and data of these plots is publicly available at https://github.com/davidedelvento/Mybrid/tree/main/data/RPiPico%2BEAITRCA6 The ADC data plots have been generated with (and manual zooming): ./parse.py -p -12 hires/p_to_f.12bit.1chan.bz2 The fake-MIDI velocities have been generated with (and manual zooming): ./parse.py -m -12 hires/p_to_f.12bit.1chan.bz2 The notes which are first and second in this plot are the third and fourth overall for that capture. The problem becomes even worse for some other notes, for which the local noise make the velocity appear with the reverse sign!

RIP

RIP I don't think I'll have the time to do it this weekend.

If any of you does the test, please let me know (you'll need a suitable SD card and some way to attach it to the Pico, as described at https://github.com/davidedelvento/no-OS-FatFS-SD-SPI-RPi-Pico ).

If you only want to play with the data, you can download it here, it is 6MB, in the same format as described at https://github.com/davidedelvento/Mybrid/tree/main/data/RPiPico%2BEAITRCA6 (note that it is in binary, not MIDI, format, unlike I did earlier).

JayKominek

RIP Do notice that with the battery the noise is indeed much better, but not as good as it seems at first sight, since the whole signal is somewhat less

With the battery on there, probably acting as a large strange capacitor, the structure of the noise is greatly reduced. That starts looking like something you can apply filtering to usefully.

Probably the signal is smaller because your battery's voltage has been pulled down by… whatever weirdness is happening. So the LED is emitting less, and Vce on the phototransistor is lower, so Ice is reduced for a given amount of light hitting it. Vref dropping because it is also connected to the battery won't account for all of that.

I'd take it as a good sign.

RIP That D-cell tests seems appropriate….

Hmm, not sure if that will be better or worse. I thought they had quite high ESR, but when I went looking for information, (1) I couldn't find it and (2) what I did find suggested it was quite a bit lower than I thought. So I look forward to hearing what happens.

RIP

Wow! The battery did it.

As you anticipated the ESR was a bit of a problem (voltage dropped to 3.05V during the test) but I was able to complete the ADC capture without attaching a computer (and thus without knowing much if it was running correctly until after the fact). Yet, look how clean it is!!

Here is a detail:

Not sure about that residual 160Hz modulation, it might be the Pico itself or it may be some external signal. Possibly even light? The setup was a bit too precarious to perform in darkness, so I had the lights on (all LEDs, as matter of fact). Pointed as far away from the hammer area as I could, but still not in darkness.

I will post the data to https://github.com/davidedelvento/Mybrid/tree/main/data/RPiPico%2BEAITRCA6 in a few moments, but I had to let you know ASAP 😃

JayKominek

RIP Not sure about that residual 160Hz modulation, it might be the Pico itself or it may be some external signal. Possibly even light?

Yeah that's a pretty odd bit of noise. One would expect 30, 60, or 120Hz noise from lights or other AC sources in one's home.

Going wildly out on a limb, do you maybe write to the SD card at some multiple of 160Hz? Or maybe accumulate some power-of-two worth of writes to the card every 160Hz? I think most flash devices do weird buffering under the hood. 250kS/s, 1/160th of a second, 1562 samples. If you, hypothetically packed the two 12 bit samples into 3 bytes, that'd be 4686 bytes, a bit more than a 4k page.

RIP

JayKominek Yeah that's a pretty odd bit of noise. One would expect 30, 60, or 120Hz noise from lights or other AC sources in one's home.

Yes! The only reason I mentioned AC lights is that I eyeballed the frequency, so it could have been 120Hz if I did it mental math wrong.

JayKominek Going wildly out on a limb, do you maybe write to the SD card at some multiple of 160Hz?

Not explicitly. The SD-card-writing is done via SPI by a library which I just peeked a bit under the hood, but I did not examine it very closely. I know that it does use IRQs.

I wanted to use IRQ for the ADC, but instead I am just sampling as fast as I can in a loop, so no particular frequency of mine (but the SPI has its own, which I set but is not fully honored: it goes somewhat slower)

JayKominek If you, hypothetically packed the two 12 bit samples into 3 bytes

That's almost what I do: 2 samples and 12 bit of elapsed time (with overflow, which I then unroll while doing data analysis). As linked in the README of the linked repo, the code for this 3-byte stuff is here and in the same repo is the library in use.

At this point I consider myself satisfied, and I'll go back to the MIDI velocity estimation based on this cleaner data. In that regard, I think that your own discussion here and @xooorx 's colevity here are already (close to) the back-of-the-envelope SG that I was asking.

In other words: if the scanning frequency is (say) a fast but comfortable 10kHz, i.e. the interval between samples is 0.1ms, then at fff the hammers move less than 1mm between samples. If we can measure the hammer position with enough spatial resolution (say 1% just to make that ballpark of 1 MIDI velocity out of a value of about 100 -- yes I know that logarithmically it will be a bit less) then we are ok in term of resolution. Eyeballing your distance-to-ADC plot, if one ignores that "dip" at close distances (which is horrible for this purpose), it feels like we do have the resolution.

The question remains that I am not sure SG will behave correctly for such a MIDI velocity computation at the upper end of the dynamics, with the approach of simply accumulating and utilizing the last ~ 23 values (as your code was doing last time I looked). If that fff speed was constant, 23 values would be about 23mm and that is about half the travel of the hammer. Obviously the speed is far from constant if such a long range, so the distance will be much less -- but the important part will be the last couple of samples only. Would a fixed-window SG filter with 23 values work as we want in these conditions? I am still not sure, but at least that's progress compared to my understanding of last week. I'll think more about it, and hopefully @xooorx and/or @luns will chime in with additional insights (and of course you, Jay, might have ones too).

« Previous Page