Category

iPXE

Wierd goings on with a Live CD and libdvdcss.so.2

I’ve been messing around trying to make a live-CD with some transcoding/ripping utilities built in to utilize some of the spare hardware I’ve got lying around. More on this later, but I’ve been reworking the guide @ http://willhaley.com/blog/create-a-custom-debian-live-environment/ with my own utilities and tools.

One problem I’ve been challenged with over the last couple of days is HandBrake-Cli bombing out with the message:

[email protected]:/mnt/Videos/Movies/dvdrip/91# HandBrakeCLI -i BHD.iso -o BHD.mkv –preset=”High Profile”
[20:41:41] hb_init: starting libhb thread
HandBrake 0.9.9 (2014070200) – Linux x86_64 – http://handbrake.fr
4 CPUs detected
Opening BHD.iso…
[20:41:41] hb_scan: path=BHD.iso, title_index=1
index_parse.c:191: indx_parse(): error opening BHD.iso/BDMV/index.bdmv
index_parse.c:191: indx_parse(): error opening BHD.iso/BDMV/BACKUP/index.bdmv
bluray.c:2341: nav_get_title_list(BHD.iso) failed
[20:41:42] bd: not a bd – trying as a stream/file instead
libdvdnav: Using dvdnav version 4.1.3
libdvdread: Missing symbols in libdvdcss.so.2, this shouldn’t happen !
libdvdread: Using libdvdcss version  for DVD access
Segmentation fault

This has been bugging me, as it worked before I converted the image to a livecd.  I wondered if it was some kind of problem with the lack of ‘real’ disk space, or a lack of memory or something like that, but nothing I could find would identify it.

Finally, I started looking into libdvdcss rather than HandBrake itself.  I think what confused me is the symbols error looks like a warning, especially given that there is a follow-on message which looks like libdvdcss is continuing.  Anyway, eventually!   I ran an md5sum on the libdvdcss.so.2 file to see if it matched a non-live machine (to a virtually identical build).

[email protected]:/# md5sum /usr/lib/x86_64-linux-gnu/libdvdcss.so.2
4702028ab20843fd5cb1e9ca4b720a72  /usr/lib/x86_64-linux-gnu/libdvdcss.so.2

N.b. libdvdcss.so.2 is symlinked to libdvdcss.so.2.1.0 in my current Debian sid based build.

On the donor machine
[email protected]:/usr/lib# md5sum x86_64-linux-gnu/libdvdcss.so.2
c9b314d9ed2688223c427bc5e5a39e6f  x86_64-linux-gnu/libdvdcss.so.2

So I’ve SCPd the source file into the live machine, checked the md5sum matched the donor machine (it did), and repeated the HandBrake job.  Lo and behold it worked!  So I’ve restreamed the two files into the filesystem and success, it just works.
So I don’t know if something funky happens when the image is created using the link, but actually its quite easy to fix once you understand the problem.

Hope this helps someone,  and I’ll be back soon with more details about building a live image, then booting it using iPXE.

SysRescueCD v3.1.2

So to kick off, we’ll start with booting SysRescueCD from http://www.sysresccd.org/.

From Wikipedia:
SystemRescueCd is an operating system for the x86 computer platform, though the primary purpose of SystemRescueCD is to repair unbootable or otherwise damaged computer systems after a system crash. SystemRescueCD is not intended to be used as a permanent operating system. It runs from a Live CD or a USB flash drive. It was designed by a team led by François Dupoux, and is based on the Gentoo Linux distribution.

For this activity, I used the download versioned v3.1.2 which I got from http://goo.gl/F36zV 

My issue was that the machine I was trying to boot from didn’t seem to have enough memory to use the memdisk/iso boot option common for most installs, which meant I had to try and boot the ISOLINUX image.

The Software:
Open the ISO in your favorite ISO opening tool (I use 7zip).  
Extract the following files into your web boot server.  I used a sub-directory called SysRescueCD

  • sysrcd.dat
  • sysrcd.md5
  • ISOLINUX/rescue32 (or 64)
  • ISOLINUX/initram.igz

Note,  I assume rescue64 is the 64 bit version of the kernel, and rescue32 is the 32bit kernel.  There is also altkrn32 and altkrn64 which are referenced in ISOLINUX as alternative kernel builds.  They all seem to work.


The Webserver Config:
This is the menu display section of the config:
item SysRescueCD32 SysRescueCD – 32bit

And this is the execution program required to boot it.

############ SYSRESCUECD ############
:SysRescueCD32
echo Starting Sys RescueCD (32bit) with default options
initrd http://boot.server/SysRescueCD/initram.igz
chain http://boot.server/SysRescueCD/rescue32 cdroot docache dodhcp setkmap=uk netboot=http://boot.server/SysRescueCD/sysrcd.dat
boot || goto failed
goto start

Note, you can change setkmap= to your preferred keyboard mapping; I’m in the UK so that is the one I use.  If you leave this option unset,  it will prompt you when you boot the server.
If you change
rescue32 to rescue64 or one of the alternate kernel images,  the same commands seem to work.  There doesn’t seem to be any difference in using netboot= or boothttp= to locate the main disk image.

Finally, I’m using a Thecus NAS as my boot webserver,  using FaJo’s Apache Webserver module.  For some reason,  whilst the initrd and kernel load perfectly well,  the image refused to boot, freezing at ‘null’ in the download.  Another Apache webserver didn’t exhibit the same condition, but its something to be aware of.  If I find the cause,  I’ll update this post.

iPXE Booting – An Introduction

If you’ve arrived at this blog,  you likely already know what iPXE booting is,  but for those who don’t,  I want to give a brief introduction to what iPXE is and how useful it is;  both for IT engineers and computer hobbyists.

So, What is it?

Quite simply,  its an open-source bootloader, primarily for the x86/amd64 machine architecture.  Its aim is to allow you to boot software and images across a network connection, using common network protocols such as tftp, http, ftp, iSCSI and AoE.

iPXE can be called over the network, started from bootable media like USB sticks or CD-Roms and even injected into the ROM of network cards where available.  This makes it an ideal tool for IT ‘fixers’ who often need to load a variety of images rather than just the standard Windows install on a machine.


A Brief History of PXE Booting.

PXE (Or pixie) booting is nothing new – in fact, pretty much all of the x86 type computers I have come across built in the last 10 years allow PXE booting from the network card, although sometimes the option needs to be turned on in the BIOS or the NIC.  Thin Clients,  VoIP phones and other ‘low touch’ devices also often support downloading configurations, software patches and other such updates over the network from a central source.   But PXE booting is not without its problems.  Its not setup to allow user interaction, meaning whatever the DHCP config is set to delivered is delivered.  Secondly, it uses tftp to transfer files – perfectly adequate for small text based config files,  but SLLLOOOOOWWWW when sending much over a few megabytes, even on fast lan connections.  Finally, PXE is only available with the functionality built into the firmware, so if it doesn’t do what you need it to,  you’re pretty much stuck with what you’ve got.

iPXE has had a bit of a convoluted development path in its reasonably short life.  It began life as ‘Etherboot‘, which started in around about 1995.  Etherboot evolved into gPXE (Nothing to do with Google) and then after a bit of a falling out within the gPXE team,  gPXE was forked by some members of that team into the iPXE we have today.  Note,  iPXE doesn’t have anything to do with Apple and their iDevices.   Both gPXE and iPXE continue to this day, doing pretty much the same thing.  However,  from my experience,  gPXE is better documented,  whilst iPXE receives more regular development activity.  And development does not happen in parallel, so you may want to check out both in case one has a feature set the other doesn’t.

A Google Tech-Talk about gPXE and Network Booting.

I’ve moved from gPXE to iPXE in 2012, mainly because gPXE wouldn’t support a NIC on a Dell machine I was trying to boot,  and had been considering making the switch at some point anyway.  iPXE resolved the issue for me,  so its now become a bit like Doc Martins were in the 80s and 90s – The boots of choice.

But why use iPXE Booting?

For me, it was a means to an end – I needed to test a motherboard and RAM combo, but didn’t have a spare hard-disk to boot from.  I also didn’t have any blank CD’s to burn a live-cd image to in order to boot from a CD drive.  Finally, I didn’t have any large (>64mb) USB thumb drives that I could write an ISO to, so I was kind of out of options.  Then I wondered if there was a way of using the small stick to chain boot something from the network and well,  it turns out there was.
I’ve stuck with network booting because once you get it working, your life as an engineer becomes much easier.  No more messing around burning CD’s, or digging through CD wallets trying to find the version of the particular utility you need.
Now I can connect up a network cable, boot from the network, choose the ISO, utility, or test tool you require from a menu,  and stand back and watch it boot from my NAS filer.  This is often quicker than waiting for a silver platter of plastic and metal spin up and be read using the power of laser light.
The process is a little complicated, but once you’ve set it up the booting piece is pretty much hands off;  the only maintenance is adding new images and updating the menu.
The way I’ve configured it is
  • PXE requests an IP Address from the DHCP Servers
  • IP Address is returned,  along with instructions on getting the iPXE ‘software’ from a TFTP server.
  • The PXE downloads iPXE from the network,  and chainloads into this.
  • iPXE re-requests an IP Address from the DHCP Servers, along with iPXE specific instructions – i.e. where to go next.
  • iPXE reads through this config and initiates a http connection to the web server containing the main iPXE config files and menu etc.
  • The end user/operator (me) chooses what to boot from the menu.
  • iPXE parses the settings supplied in the menu, downloads the necessary components (again, over http) and boots the image.
gPXE took me about a day to get working,  because of the need to install and configure a DHCP server with various options.  Because this is a home network, I had to create this new by learning ISC DHCP,  plus I took the opportunity to install ISC BIND as a DNS server – further learning required.  Note,  you don’t *need* these for iPXE to work, but it makes life easier in the long run.   I may blog more about setting up iPXE at some point,  but there are plenty of documented guides on how to do it.   

And what is the point of this blog?

So, as I’ve already mentioned above,  iPXE doesn’t have *that* great a set of documentation,  particularly in regard to configuring images to boot.  Most of these comes through gleaning facts from various sources,  putting them together through trial, error and a little bit of prior experiencing and seeing if it works.   Whilst I manage to get things to boot,  I think its useful to share solutions with the wider community to make things easier to use.  This is particularly true when trying to convert Linux live images into an iPXE boot set up, where you don’t necessarily want to or can’t load an entire ISO into memory.  Hopefully others will find this a useful resource,  and I’m going to try and use it as a library of knowledge on my own experiences of network booting.