All Posts By

verdegemn

Big Data Conversations.

I’m not sure people understand big data and mining the information held within. I’ll summarise a conversation I had today.

Me: Can I have that data-set please?
Other Person (OP): Its got over 20 million records in it, what do you want to know?
Me: I am thinking about x, and think your data set may answer some questions.
OP: What exactly are you looking for?
Me: I don’t really know, until I’ve seen the data and what information it holds.
OP: How do you know my data-set has the information you need?
Me: I don’t. but its the best chance I’ve got.

and so on and so forth…

I think sometimes big data mining is a bit like mineral mining. You can take samples and investigate indicative factors, but until you take hold of your pick-axe, you’ll never know exactly what is down there. Hopefully I’ll get access to the output and see what can be discovered from it. I am already thinking about visualisation techniques to find the shiny nuggets of data held within.

A little Comic Relief

The annual or bi-annual charity appeals are always fun, but also somewhat predictible as to what we’ll end up seeing.  So, whilst I urge you to donate,  you can also have a bit of fun whilst watching.

So here’s the rules of the drinking game we’re playing by:


The rules are simple:
Define your own measure,  be it a shot,  finger’s width or even a whole glass.

One Measure
– A disease is mentioned
– An African Child is seen with a fly on its face
– Every time a celebrity does something ‘exciting’
– A giant cheque is produced
– A celebrity holds an African child
– A celebrity crys
– The Phone Number is read out
– Man dressed as a woman, or a woman dressed as a man

Two Measures
– The total so far is read out
– We see the phone call takers up BT Tower
– When the guest presenters change
– When the presenters look confused because they don’t know what they’re going to next
– If we’re shown a picture of BBC TV Centre

Three Measures
– When the presenter runs his/her fingers along the screen as the number is read out.
– When someone mouths the number at the back of shot
– When someone makes the ‘phone call’ symbol
– News Presenters doing something ‘wacky’
– When an African child is made to wear a red-nose

Four Measures
– Wogan or Pudsey Appears

Penalties:
Penalties require a CR donation
– Spillage – £1
– Fall Over – £1
– Vomit – £5
– Pass-out – Whatever you can shake out of their wallet/purse.

Remember Kids, Drink Responsibly,  as little African Kids often can’t.   Give Generously!

iPXE CloneZilla

CloneZilla is a linux toolset that allows you to clone either a partition or whole disk to another location;  either a connected storage device,  or remotely to the network.   This is a great tool for imaging systems before you work on them and lets you have a copy in case the worst should happen.    It has a variety of bundled tools in order to read from most of the popular filesystems in use,  falling back to DD to copy each disk sector if you’re using some obscure or proprietorial filer.   This is the FOSS alternative to Norton Ghost!

The great thing about CloneZilla is that its quick and easy to get it booting via iPXE,  so is worth investing a small amount of time in setting up so that you have it ready to go should you need it.

These instructions are based on release clonezilla-live-20121217-quantal.iso which seems to be versioned 2.0.1-15.  

Download the ISO from the CloneZilla site.  Use 7zip or your favourite image opening tool to open the ISO.  You need to extract the following files:

  • vmlinuz
  • initrd.img
  • filesystem.squashfs

and put them onto your boot webserver.  In this example,  I have created a folder called CloneZilla.

############ CloneZilla ############
:Clonezilla
echo Starting CloneZilla with default options
kernel http://boot.server/CloneZilla/vmlinuz
initrd http://boot.server/CloneZilla/initrd.img

imgargs vmlinuz boot=live config noswap nolocales edd=on nomodeset ocs_live_run=”ocs-live-general” ocs_live_extra_param=”” ocs_live_keymap=”” ocs_live_batch=”no” ocs_daemonon=”ssh” usercrypted=Kb/VNchPYhuf6 ocs_lang=”” vga=788 nosplash noprompt fetch=http://boot.server/CloneZilla/filesystem.squashfs
boot || goto failed
goto start


And that is really about it! You’ll notice we pass a few arguments which set various options. The most important option is the ‘fetch=’ command which tells the image where to download the main file system from. The other option I set was ‘usercrypted=’ which uses the Linux mkpasswd command to set the root password on boot – in this example iloveclonezilla.

A really easy one this week, but one worth trying. I’m fighting to get Backtrack5 booting over iPXE without using the ISO method, but this is proving troublesome. I think the image simply isn’t able to cope with being booted from a http network source.

Sugru – A Brief Update

I just wanted to provide a brief update on my thoughts of Sugru.  Its a wonderful product, ideal for fixing and personalising things.  However,  the biggest bugbear of all is the shelf-life.

Unlike duck/duct tape,  superglue,  epoxy resin, putty  and other more commonly known fixing materials and methodologies,  Sugru ‘sets’ after about 6 months,  whether you’ve opened it or not.  This means that one of the big bags I had became useless as I wasn’t able to use the sachets before they’d all set.

And because Sugru isn’t available in most shops,  you can’t just pop and get some more – you have to wait for the postman to bring it for you.  This is fine if you have a non-urgent fix, but when you need to do something straight away,  you either have to ensure you have some fresh Sugru in,  or find an alternative option.  Its often the latter.  So reader beware!

One final point,  Sugru reckon if you keep it in the fridge, it will keep for 18 months;  I’ll have to pick some up and try it.

SysRescueCD v3.1.2

So to kick off, we’ll start with booting SysRescueCD from http://www.sysresccd.org/.

From Wikipedia:
SystemRescueCd is an operating system for the x86 computer platform, though the primary purpose of SystemRescueCD is to repair unbootable or otherwise damaged computer systems after a system crash. SystemRescueCD is not intended to be used as a permanent operating system. It runs from a Live CD or a USB flash drive. It was designed by a team led by François Dupoux, and is based on the Gentoo Linux distribution.

For this activity, I used the download versioned v3.1.2 which I got from http://goo.gl/F36zV 

My issue was that the machine I was trying to boot from didn’t seem to have enough memory to use the memdisk/iso boot option common for most installs, which meant I had to try and boot the ISOLINUX image.

The Software:
Open the ISO in your favorite ISO opening tool (I use 7zip).  
Extract the following files into your web boot server.  I used a sub-directory called SysRescueCD

  • sysrcd.dat
  • sysrcd.md5
  • ISOLINUX/rescue32 (or 64)
  • ISOLINUX/initram.igz

Note,  I assume rescue64 is the 64 bit version of the kernel, and rescue32 is the 32bit kernel.  There is also altkrn32 and altkrn64 which are referenced in ISOLINUX as alternative kernel builds.  They all seem to work.


The Webserver Config:
This is the menu display section of the config:
item SysRescueCD32 SysRescueCD – 32bit

And this is the execution program required to boot it.

############ SYSRESCUECD ############
:SysRescueCD32
echo Starting Sys RescueCD (32bit) with default options
initrd http://boot.server/SysRescueCD/initram.igz
chain http://boot.server/SysRescueCD/rescue32 cdroot docache dodhcp setkmap=uk netboot=http://boot.server/SysRescueCD/sysrcd.dat
boot || goto failed
goto start

Note, you can change setkmap= to your preferred keyboard mapping; I’m in the UK so that is the one I use.  If you leave this option unset,  it will prompt you when you boot the server.
If you change
rescue32 to rescue64 or one of the alternate kernel images,  the same commands seem to work.  There doesn’t seem to be any difference in using netboot= or boothttp= to locate the main disk image.

Finally, I’m using a Thecus NAS as my boot webserver,  using FaJo’s Apache Webserver module.  For some reason,  whilst the initrd and kernel load perfectly well,  the image refused to boot, freezing at ‘null’ in the download.  Another Apache webserver didn’t exhibit the same condition, but its something to be aware of.  If I find the cause,  I’ll update this post.

iPXE Booting – An Introduction

If you’ve arrived at this blog,  you likely already know what iPXE booting is,  but for those who don’t,  I want to give a brief introduction to what iPXE is and how useful it is;  both for IT engineers and computer hobbyists.

So, What is it?

Quite simply,  its an open-source bootloader, primarily for the x86/amd64 machine architecture.  Its aim is to allow you to boot software and images across a network connection, using common network protocols such as tftp, http, ftp, iSCSI and AoE.

iPXE can be called over the network, started from bootable media like USB sticks or CD-Roms and even injected into the ROM of network cards where available.  This makes it an ideal tool for IT ‘fixers’ who often need to load a variety of images rather than just the standard Windows install on a machine.


A Brief History of PXE Booting.

PXE (Or pixie) booting is nothing new – in fact, pretty much all of the x86 type computers I have come across built in the last 10 years allow PXE booting from the network card, although sometimes the option needs to be turned on in the BIOS or the NIC.  Thin Clients,  VoIP phones and other ‘low touch’ devices also often support downloading configurations, software patches and other such updates over the network from a central source.   But PXE booting is not without its problems.  Its not setup to allow user interaction, meaning whatever the DHCP config is set to delivered is delivered.  Secondly, it uses tftp to transfer files – perfectly adequate for small text based config files,  but SLLLOOOOOWWWW when sending much over a few megabytes, even on fast lan connections.  Finally, PXE is only available with the functionality built into the firmware, so if it doesn’t do what you need it to,  you’re pretty much stuck with what you’ve got.

iPXE has had a bit of a convoluted development path in its reasonably short life.  It began life as ‘Etherboot‘, which started in around about 1995.  Etherboot evolved into gPXE (Nothing to do with Google) and then after a bit of a falling out within the gPXE team,  gPXE was forked by some members of that team into the iPXE we have today.  Note,  iPXE doesn’t have anything to do with Apple and their iDevices.   Both gPXE and iPXE continue to this day, doing pretty much the same thing.  However,  from my experience,  gPXE is better documented,  whilst iPXE receives more regular development activity.  And development does not happen in parallel, so you may want to check out both in case one has a feature set the other doesn’t.

A Google Tech-Talk about gPXE and Network Booting.

I’ve moved from gPXE to iPXE in 2012, mainly because gPXE wouldn’t support a NIC on a Dell machine I was trying to boot,  and had been considering making the switch at some point anyway.  iPXE resolved the issue for me,  so its now become a bit like Doc Martins were in the 80s and 90s – The boots of choice.

But why use iPXE Booting?

For me, it was a means to an end – I needed to test a motherboard and RAM combo, but didn’t have a spare hard-disk to boot from.  I also didn’t have any blank CD’s to burn a live-cd image to in order to boot from a CD drive.  Finally, I didn’t have any large (>64mb) USB thumb drives that I could write an ISO to, so I was kind of out of options.  Then I wondered if there was a way of using the small stick to chain boot something from the network and well,  it turns out there was.
I’ve stuck with network booting because once you get it working, your life as an engineer becomes much easier.  No more messing around burning CD’s, or digging through CD wallets trying to find the version of the particular utility you need.
Now I can connect up a network cable, boot from the network, choose the ISO, utility, or test tool you require from a menu,  and stand back and watch it boot from my NAS filer.  This is often quicker than waiting for a silver platter of plastic and metal spin up and be read using the power of laser light.
The process is a little complicated, but once you’ve set it up the booting piece is pretty much hands off;  the only maintenance is adding new images and updating the menu.
The way I’ve configured it is
  • PXE requests an IP Address from the DHCP Servers
  • IP Address is returned,  along with instructions on getting the iPXE ‘software’ from a TFTP server.
  • The PXE downloads iPXE from the network,  and chainloads into this.
  • iPXE re-requests an IP Address from the DHCP Servers, along with iPXE specific instructions – i.e. where to go next.
  • iPXE reads through this config and initiates a http connection to the web server containing the main iPXE config files and menu etc.
  • The end user/operator (me) chooses what to boot from the menu.
  • iPXE parses the settings supplied in the menu, downloads the necessary components (again, over http) and boots the image.
gPXE took me about a day to get working,  because of the need to install and configure a DHCP server with various options.  Because this is a home network, I had to create this new by learning ISC DHCP,  plus I took the opportunity to install ISC BIND as a DNS server – further learning required.  Note,  you don’t *need* these for iPXE to work, but it makes life easier in the long run.   I may blog more about setting up iPXE at some point,  but there are plenty of documented guides on how to do it.   

And what is the point of this blog?

So, as I’ve already mentioned above,  iPXE doesn’t have *that* great a set of documentation,  particularly in regard to configuring images to boot.  Most of these comes through gleaning facts from various sources,  putting them together through trial, error and a little bit of prior experiencing and seeing if it works.   Whilst I manage to get things to boot,  I think its useful to share solutions with the wider community to make things easier to use.  This is particularly true when trying to convert Linux live images into an iPXE boot set up, where you don’t necessarily want to or can’t load an entire ISO into memory.  Hopefully others will find this a useful resource,  and I’m going to try and use it as a library of knowledge on my own experiences of network booting.

We Canna do it Captain, we don’t have the Power.

Last night, we experienced an area wide power cut.   Despite being in modern 21st century,  this is nothing unusual – they seem to occur 3-4 times per year, with an average length of about 20mins.

Many people in the area are fully prepared,  with battery powered lanterns, torches and the traditional candles and matches to hand.  But after you’ve provided yourself a bit of light (when it happens at night), you find yourself sitting there, wondering what to do now the TV, Radio, Computers have all gone off.   And of course, whilst your broadband has almost certainly gone off, your smartphone falls back to the mobile network for its internet which more often than not keeps working.  Anyone using social networks via their phone can tweet for facebook that the power has gone out,  and start comparing notes as to how far it stretches.  Most people wouldn’t bother to phone their power supplier to let them know about the outage, as I think the assumption is that they can detect it and start resolving it ASAP.

But do they?

The previous supplier of electricity, Central Networks used to have a map that you could click on and see where they knew there were faults.  Western Power Distribution http://goo.gl/JjuKU , the new incumbent do not.  Now,  the information used to be available to the suppliers, so where has it gone now?  Do they no longer have the systems to collect this data from the grid,  or was it collated from their CRM systems?  Maybe they feel that it is some kind of sensitive data so best not to share it,  or simply that the bosses do not think this information is valuable to its client base.
I’m singling out Western Power, but a quick search of the power companies listed by the National Grid (http://goo.gl/7XSpX) seem to highlight only 
Northern Power Grid (http://goo.gl/OrPvNand
Electricity Northwest (http://goo.gl/lvIwR)
seem to be willing and/or able to provide this information.   And good on them too.  Not only do ENWL show current unplanned outages, but also future planned work.   

So my point is,  if they can do it,  why can’t other utility providers provide outage information.  Or,  maybe its something that the National Grid can do – after all, they provide a live view of the demand (http://goo.gl/A3tv6),  but can they see deep enough into the local grids to see the outages. 

Power cuts happen – its a fact of life.  But how long will it take suppliers to embrace the communication power of the Internet to get information out to its customers.  For now,  I guess we’ll just have stick with searching twitter for #PowerCut

Do you, Sugru?

How many times have you bought a fantastic product that is perfect in every way, right up until the moment it breaks.  Or you’ve got something that performs a function, but isn’t quite right.  Historically, you’d end up going out and buying a replacement – something that may still not do the job.   Well, I’ve discovered a product which may help you reduce your ‘throw away and replace’ approach with a ‘make do and mend’ attitude.  Only without the make-do.

Sugru is a product I heard about in the late summer of 2010 and it immediately peaked my interest.  A product invented in the UK by an Irish woman,  it is a mouldable, self-curing silicone with lots of really nice features.  When you get it from its sachet, it has the consistency of too-warm bluetak – a sticky clay like substance that can be moulded into any shape you choose.  It initially sets within 30minutes then cures within 24hours to whatever shape you’ve moulded it from. When it sets, it turns into something with the consistency of hard-rubber.  It retains a slightly springy texture, the sort of resistance of a block-eraser.  It also self-adheres to many materials like glass, metal  & plastics, works between –60c and +180c, is thermally insulating,  waterproof, even dishwasher safe!  Sugru Sample

I bought some packs for Xmas and gave friends and family a sachet each to try, and kept some for myself.  This weekend, I went on a bit of a Sugru session because I had two things to try.   First of all, my car stereo security faceplate had broken – one of the retaining clips broke off with plastic fatigue.  However, I found that a bit of paper wedged in the slot  would hold it in place.  So I applied a bit of Sugru to the faceplate, left it for about 8hrs then tried it.  It worked!  Even better, the clip that holds the faceplate in had moulded a little indentation in the Sugru which held it even more securely.  Another 12hrs of curing, and you wouldn’t know that the thing was broken in the first place.  That’s £30 saved on a replacement facia or £100+ on a replacement stereo.

The second usage for it was to adapter a mobile phone holder.   I’ve got a small phone-chair which is designed to rest a mobile phone in and keep the screen visible. However, because of its shape, I couldn’t put the phone in its naturally portrait orientation and charge it at the same time – it would have to be landscape with the cable at the side, which was OK, but not all apps support rotation. So I moulded two Sugru ‘ears’ at the front of the chair, let it set, and now the results can be seen right. Its not perfect (my craft skills are somewhat lacking) but it certainly does the job.  Only a few pounds saved but it does the job perfectly.

Sugru seems like a brilliant product – the WD40 equivalent of a physical medium.  However, there are problems with it:- 

  • it seems expensive – about 95p per 5g sachet (which is what I used here).  However, its still cheaper than buying new!  
  • It can be a bit of a pain to handle – it seemed difficult to get small pieces of it to stick to the object I wanted it to adhere to, rather than my fingers.  Plus, whilst it is classified as “not dangerous” it may cause an allergic reaction.  Its probably no more dangerous than any other chemical based material.
  • It currently only has one cured state.  Sugru say that they can mix different formulations, but currently only offer this one.
  • Thinking of things to do with it.  With only a 6month ‘best before’ date, I will have to find things to use it on in the next few months.  However, there are lots of ideas on the Sugru website including repairing broken shoes, making tent pegs softer so that they could be hand inserted, and forming a bespoke golf-club handle.

I would seriously recommend picking up a pack (£6.50 for a 6 pack, £11.50 for a 12 pack) and trying it out on something – really clever stuff.

Adventures with the Omnima LCD Panel

I have recently picked up an Omnima LCD panel, a 3.5” QVGA (320×240) USB powered LCD panel with an SDCard slot and the ability to control it like a serial device over the USB port using the included cable.

Having an ARM7 based processor and a 2D graphics engine, it promises to be a great little versatile display panel for outputting information not requiring a full screen (sensor information e.t.c, news feeds, that sort of thing).  Plus, many of the LCD panels available on the internet either require an additional controller or some curious interfacing to be able to drive it.  For not a lot of cash, this prebuilt solution seems to be an ideal choice.

So that’s the good bit – the bad bit is that there doesn’t seem much support for it.  The supplier forums are quiet at best (and not accepting new sign-ups) and there doesn’t appear to be much documentation other than the PDF’s on the Omnima website/forum.  I suspect there may be more available if you buy the SDK tools, but these are expensive to say the least.  Plus, I don’t have that much interest in writing new code for the screen CPU because I believe the inbuilt functionality is enough, and I’m not skilled in C++ development.

Anyway, the screen (version 3 as shown on the website) itself works with LCDSmartie (I’m using v 5.4.1) and the DLL included in the forum worked after a fashion.  The OmnimaLCD.cmd file included in the ZIP file isn’t quite right.  The standard command ends up rendering every serial command to the screen, rather than the result of the command.  The below command sets it up properly.

#@Term MW Off
#@Term FW Off
#@Cls
#@FrColor 0 255 0 0
#@FloatWin Open 10 10 240 320
#@Line MW 10 10 50 10
#@Line MW 10 50 10 10
#@Line MW 10 190 10 230
#@Line MW 10 230 50 230
#@Line MW 270 10 310 10
#@Line MW 310 10 310 50
#@Line MW 270 230 310 230
#@Line MW 310 190 310 230

This turns off any terminal command writing to the screen, creates a black box on the screen, creates a float window to render the text in, and draws some nice green lines at the edges screen which I think look quite nice.  One thing to be aware of,  you can use 4 x 20 screen setting as recommended, and you can use 4 x 40 setting.  However, on the latter, if the text exceeds the screen width, you get text wraparound where it runs around onto the left of the screen.  Not so bad if you don’t have scrolling text and format your text lines appropriately.  Also, the on screen left position seems to be set in the DLL file – I had a bash at editing what looked like the command in a HEX editor, but to no avail.

A note really for myself – the line plotting code consists of x horizontal poz, x vertical poz, y horizontal poz, y vertical poz. 

The one thing I need to work out now is how to render pictures to the screen – apparently supported, but I’m not getting much joy with it.  The graphics processor apparently supports JPEG pictures, but when I try and load it, the unit locks up and needs a power cycle.  The documentation suggests I need to use the “oimage tool” to convert from JPG, PNG e.t.c to a compatible format:-

To generate files suitable for loading using FileToSSD you can make use of the oimage tool. This tool can load all popular image file formats such as jpg, gif, bmp and more, and save the file in the SSD compatible format.

I’ve Googled the heck out of oimage and SSD compatible format, but can’t find anything of relevance – if anyone has any clues, please leave me a note.  I’ve asked Omnima directly, but not yet had a response.  Given that the device is basically a mini picture frame, it would be nice to be able to load and display pictures as well as vector graphics, especially as the latter can be quite slow (I would guess at 500ms per line) on the LCDSmartie example above.  The documentation suggests that picture loading can be quite quick, but we’ll see. 

Finally, id like to get it working in LCD4Linux – I cant see it being too difficult if its just got to send information via text strings, but that’s a project for another time.

Dreambox Sky Bouquets on Astra 28.2

I see a fair number of requests for people looking for the bouquets for their dreambox.  The one I published is well out of date, because I use some excellent ones provided by user B16MCC over at the satellites.co.uk forum.  Have a look at THIS thread, sign-up and say that I sent you 🙂