[Resolved] Latest change killed gqrx

[dk5ras]

Hi,

after updateing bladeRF yesterday gqrx stopped working...this is the output:

gr-osmosdr v0.1.1-1-g37b09bc5 (0.1.2git) gnuradio v3.7.4git-74-gf5dd2ac0
built-in source types: file fcd rtl rtl_tcp hackrf bladerf rfspace
Using Volk machine: avx_32_mmx_orc
gr-osmosdr v0.1.1-1-g37b09bc5 (0.1.2git) gnuradio v3.7.4git-74-gf5dd2ac0
built-in source types: file fcd rtl rtl_tcp hackrf bladerf rfspace
[bladeRF source] Using nuand LLC bladeRF #0 SN 8efd...ad57 FW v1.6.1 FPGA v0.0.3
[INFO] Clamping bandwidth to 28000000 Hz
[bladeRF source] bladerf_sync_rx error: Operation timed out
[bladeRF source] bladerf_sync_rx error: Operation timed out
ras@ubuntu:~$


By the way, I completely rebuilt gr-osmosdr, gr-iqbal and gqrx, with the same result as above.


Any ideas about this?

Ralph.

[jynik]

Hi Ralph,

Sorry for the trouble -- a bunch of changes in a "dev-libbladeRF_v0.15.0" branch were merged to master before I had finished testing and reviewing them as the result of urgent problems elsewhere and a communication breakdown. I'll be pushing even harder now for such changes to remain staged in a "dev" or "next" branch until folks like yourself can help beat up new changes and report issues, before they reach master. Given that more and more people are using the code, I'd really like to see things reach master only after some good reports across OSes and distros.

The changes consist of modifications to the underlying async interface to address contention and thread safety issues. These issues were identified by the people working on OpenBTS. The problems basically involved in RX starvation because TX callbacks were blocking. In fixing this, I too have seen a regression in the performance in gr-osmosdr-based programs like GQRX, with libbladeRF being the guilty party, but have been AFK over the last few days and unable to address it yet.

Until I can bring a fix in, one workaround with GQRX (or any other gr-osmosdr-based applications that might not mind the extra latency), you could try increasing the number of buffers used. At 40 Msps, a 28 MHz bandwidth, and the largest FFT and frame rate options, I was able to achieve my "usual" performance by using 128 buffers.

You can do this in the "device parameters" configuration via: bladerf=0,buffers=128

Update: Even with the above change and my CPU frequency scaling governer set to performance, I'm seeing this failure mode pretty consistently on my laptop. I'll be working on this throughout the week, and will post back comments/info here as I can. In the meantime, it might be safer to check out the the repo at the libbladeRF_v0.14.0 tag if you're going to be using GQRX pretty heavily. Stay tuned...

- Jon

[dk5ras]

OK, no problem, so the issue is known and worked on I just went back to an earlier version anyway...

Ralph.

[blagoja]

I have the same problem with GQRX after the updating bladeRF in Ubuntu 13.10.

Didn't know that the update of bladeRF could cause problems with gr-osmosdr-based applications including the GQRX.

I'm not using GQRX pretty heavily, but would like to have OpenBTS completly functional on my bladeRF

[jynik]

Didn't know that the update of bladeRF could cause problems with gr-osmosdr-based applications including the GQRX.

Well...it will only cause problems when using the bladeRF with GQRX. gr-osmosdr is the layer between GNU Radio and libbladeRF, which presents a device-agnostic source & sink interface. Hence, by breaking something in libbladeRF, gr-osmosdr-based applications (using the bladeRF) will also be affected.

[jynik]

If anyone feels like playing around a bit to help gather some info -- it would be helpful for me to hear whether increasing the number of buffers used by gr-osmosdr via 'buffers=32', 'buffers=64', 'buffers=128' improves the current situation. (Could you sound off on your setup + Distro when reporting back, just for my own curiosity?)

I'm currently listening to a narrowband FM signal, with the bladeRF sampling 28Mhz of BW @ 40Msps, with the FFT size set to 16384 and a 60 fps refresh. On an i7-3630QM running at 2.4 GHz, I'm looking at ~50% CPU usage on every core. Once it's up and running, it seems stable. However, it seems like the failures occur when starting up. (As an amusing aside, it seems if my laptop fan is really working, I'm golden. If it slows down, there's a good chance I'll hit the failure mode.)

With some recent changes to the async interface, there's a bit more logic in the async callbacks. However, this sort of regression seems extreme for the relatively little code added. Hopefully I'll find and squash a silly bug. In the meantime, here's a few values to play around with in the GQRX "device string" (this is actually the gr-osmosdr device args string):

• bladerf=0,buffers=32
• bladerf=0,buffers=32,transfers=31
• bladerf=0,buffers=64,transfers=32
• bladerf=0,buffers=128,transfers=32
• bladerf=0,buffers=256,transfers=32
• bladerf=0,buffers=512,transfers=32
• bladerf=0,buffers=1024,transfers=32

Update: Opened an issue for this on the tracker, and will update status there as things unfold and/or close it when a fix is in: Issue #232

[jynik]

Update: The fix referenced here has been merged to master. The dev-issue_232 branch has been removed, and issue 232 has been closed.

Hi all,

A patch has been pushed to a dev-issue_232 branch. On my laptop (the i7 previously mentioned) with cpu frequency scaling disabled, I'm happily running at the max sample rate, bandwidth, FFT size, and frame rate while listening to the audio from a narrowband FM signal, with no overruns. My device string while doing this is: bladerf=0,verbosity=debug,buffers=128,transfers=32

The verbosity=debug will enable the printing of a debug message when an overrun occurs. It would look something like this: [DEBUG] RX overrun @ buffer 5.

Could I grab a few folks to beat this up and report back? Below is the gist of building and installing with this patch:

Code: Select all

# (1) Check out the branch with the patch
~ $ cd ~/projects/bladeRF
~/projects/bladeRF $ git pull
~/projects/bladeRF $ git checkout -b dev-issue_232 remotes/origin/dev-issue_232

# (2) Just for thoroughness, do a clean build. This is technically overkill here,
# but it's generally nice to do in case configuration options have changed
~/projects/bladeRF $ cd host
~/projects/bladeRF/host $ rm -rf build && mkdir build && cd build
~/projects/bladeRF/host/build $ cmake ../ && make && sudo make install

# (3) Test GQRX!

# (4) Switch back to master and delete the dev branch
~/projects/bladeRF/host/build $ git checkout master
~/projects/bladeRF/host/build $ git branch -D dev-issue_232

# (5) Repeat step (2) to rebuild master

Some general tips:

• RX overruns will result in discontinuities. You might notice these in the frequency domain as a sudden wide-band spike in the noise floor. Use the verbosity=debug to get some output noting when overruns occur.
• When you find you're starting to push the limits of your machine, try incrementing the number of buffers with values like 64, 128, 256. For those values, I'm seeing transfers=32 work nicely

buffers vs. transfers:
This isn't really obvious if you're not in the code, so hopefully this makes sense...

• buffers=<n> Denotes the total number of fixed-size sample buffer to allocate and use. In short, this corresponds to how much memory you want to reserve for samples. A higher value will help avoid some transient overruns that might occur if your CPU becomes temporarily busy, but induces more latency.
• transfers=<n> Denotes the max number of how many of the above buffers are scheduled to be filled ("in-flight") at any given moment in time. This allows us to have the bladeRF continue filling buffers while we're in the process of figuring out which one to submit next.