Back to Linux


After spending a long time trying to do big cases of my SKA1-LOW calibration analysis, So I decided that I needed some more compute power. After a lot of investigation, I decided on an eight core Infinity Xtreme Pro from CyberPower PC, along with a GeForce 1080, and 64GB memory, plus 256 GB SSD. I did look at a Mac Pro but it's twice as much and uses AMD graphics cards, which I don't want to program.

I ordered it about 10pm Wednesday and then it was in my hands Friday just after noon.

I'd forgotten what a misery installation of desktop Linux is. Back in 2004, I got my first Mac and never looked back, at least for my personal machine. To cut a long, tedious story much shorter than it felt, I should have bought a vanilla graphics card to run the monitor. To run the GeForce as a graphics card without the proprietary Nvidia drivers,you need a brain dead monitor. I duly got my own TV from Australia and that worked fine and then I could install the Nvidia drivers. I had a working system. Installed all the things that I needed - python, etc. Also set up a way to get to the computer from my MBP. All of this required searching some long-closed rooms in my own memory palace.

Here's the good part. 8 cores each of which can run 2 threads means that I can 16-way distribute my low-calibration calculations. Thanks to the package pymp, all I had to do was re-write a loop from this:


for station in range(nstations): print('Calculating station %d to all other stations' % (station)) for noll in range(nnoll): for source in range(sources.nsources):
… stuff here … A[noll, :, 0] += numpy.real(vphasor) A[noll, :, 1] += numpy.imag(vphasor)



to this:


for station in range(nstations): print('Calculating station %d to all other stations' % (station)) A = numpy.zeros([nnoll, nstations, 2]) with pymp.Parallel(4) as p: for noll in p.range(nnoll): for source in range(sources.nsources):
… stuff here …
with p.lock: A[noll, :, 0] += numpy.real(vphasor) A[noll, :, 1] += numpy.imag(vphasor)


This worked on my MBP first time. Took some experimentation to figure out which order of loops worked fastest and then I was done. Put into github, retrieved on the Linux machine, and changed 4 to eight and it again worked first time. The big machine, which I've called sheldon, can run about 15 of these processes at one. There is a batch processing queue built in.

The upshot is that I can now run the very biggest case that I need to.

So python is great for this work, pymp is very impressive, Linux is good for this kind of work but installation of desktop Linux is a trial. Anyway, I'm pleased. I succeeded as my own IT manager.