I had built a NAS on my Shuttle using CentOS Linux and running RAID1. This little device was doing everything. NAS, DLNA Streaming Media, Digital Video Recording and BitTorrent. I then added an IMAP email server. At this point I found my self booting up a Windows XP Desktop just to run a browser to connect to the NAS to all access all these functions.
It seemed a bit pointless having a Windows Desktop PC, so I installed Gnome Desktop on the CentOS machine and just turned on the screen when I wanted to access any of the NAS functions. In addition to this I have the option of accessing the NAS from anywhere in the house via the laptop. Or even any machine on the Internet.
After this test, I decided to rebuild the NAS using Ubuntu to get a later kernel, more supported applications, and the latest graphic Gnome environment rather than use CentOS. Ubuntu Linux is great. It just works, picks up all the hardware and installs like a dream. Gnome is a very nice uncluttered desktop environment.
The on board VIA Chrome9 video card Linux drivers were not really up to the job so I had to add a dedicated graphics card. The new Shuttle spec looks like this:
|
Shuttle SK22G2, proprietary VIA K8M890CE + VIA VT8237R+ chipset Motherboard |
|
|
AMD Athlon 64 3500+ Socket AM2 |
|
|
LeadTek WinFast PX7300 GT TDH PCI-express Graphic Card |
|
|
2Gb DDR2 667MHz RAM |
|
|
500Gb ST3500630AS |
|
|
500Gb ST3500630AS |
Note the lack of DVD+RW drive. I just find that these days I hardly ever use CD or DVDs. For transferring data I use 1Gb to 8Gb thumb drives. For backups, I use a USB Hard Drive. All other content is downloaded or digitally recorded and then streamed. I have a USB DVD+RW I can attach if I need it.
Installing Ubuntu is easier than Windows. Put the CD and it just works! All the hardware is recognised.
I chose a RAID1 layout. This time I did not bother with Logical Volume Manager (LVM). LVMs don't give much advantage on machines with a single mirrored disk in.
After the install the file system layout looks like this:
[root@vader ~]# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/md0 481722696 265264388 192180888 58% / varrun 517172 2840 514332 1% /var/run udev 517172 84 517088 1% /dev devshm 517172 584 516588 1% /dev/shm |
The RAID layout looks like this:
root@vader:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda5[0] sdb5[1]
2835328 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
485548416 blocks [2/2] [UU]
|
And the physical discs:
root@vader:~# fdisk -l Disk /dev/sda: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x8baf2c2f Device Boot Start End Blocks Id System /dev/sda1 * 1 60448 485548528+ fd Linux RAID autodetect /dev/sda2 60449 60801 2835472+ 5 Extended /dev/sda5 60449 60801 2835441 fd Linux RAID autodetect Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xb62a89b4 Device Boot Start End Blocks Id System /dev/sdb1 1 60448 485548528+ fd Linux RAID autodetect /dev/sdb2 60449 60801 2835472+ 5 Extended /dev/sdb5 60449 60801 2835441 fd Linux RAID autodetect |
A nice clean uncluttered Gnome desktop.
The GUI is fast and responsive and the virtual desktops are fantastic.
Interestingly Linus Torvalds does like Gnome:
This “users are idiots, and are confused by functionality” mentality of Gnome is a disease. If you think your users are idiots, only idiots will use it. I don’t use Gnome, because in striving to be simple, it has long since reached the point where it simply doesn’t do what I need it to do. Please, just tell people to use KDE.
Which just goes to show people who write kernels are not the people who should be writing GUIs. If the GUI does not do what some egg head geek needs then just use the command line! Wake up and smell the coffee Linus, most users are idiots, hey most people are idiots. Point and click. Gnome is what is needed to compete with Windows.
To stream video and music from the NAS to the PS3 I installed Twonky Media Server.
UK Digital TV Freeview or BitTorrent downloads are saved to the NAS and then streamed wirelessly to the PS3 to watch on HD LCD TV.
I wanted a headless BitTorrent download program that I could use a web browser to connect to and administer torrents. I downloaded the latest BitTornado code and wrote my own PHP front end:
The PHP front end executes btdownloadheadless.py which forks and downloads the torrent in the background:
HOME=/data/video/NEW/; export HOME; /data/vader/web/torrent/BitTornado/btdownloadheadless.py /data/video/NEW/.transfers/_isoHunt__Angel___Complete_Season_4___DVD_Rip.torrent --saveas "/data/video/NEW//incoming/Angel - Complete Season 4 - DVD Rip" --display_interval 1 --max_download_rate 0 --max_upload_rate 0 --max_uploads 4 --minport 49160 --maxport 49300 --rerequest_interval 1800 --super_seeder 0 --max_connections 40 | /data/vader/web/torrent/log.pl /data/video/NEW/.transfers/_isoHunt__Angel___Complete_Season_4___DVD_Rip.torrent.stat 1> /data/video/NEW/.transfers/_isoHunt__Angel___Complete_Season_4___DVD_Rip.torrent.error 2> /data/video/NEW/.transfers/_isoHunt__Angel___Complete_Season_4___DVD_Rip.torrent.error & echo $! |
The hard part is dealing with btdownloadheadless.py status output. Every x seconds it echos the torrent status to stdin. I got round this by piping it to my perl program log.pl which reads the stdin and writes the latest info to a torrent.stat file.
Using the PHP Web GUI I can connect to my downloads anywhere in the world over SSL on the Internet and start, stop or even add more torrents. The content is the wirelessly streamed to my PS3 in the lounge to watch.
Next I added Digital Video Recording (DVR/PVR) capabilities to the NAS. I picked up a Hauppauge WinTV Nova-T SE Digital Video Broadcast Terrestrial (DVB-T) USB Stick for 25 squids at PC world.
Hauppauge do not supply any Linux drivers for their kit so I had to rely on OpenSource and the work being done over at www.linuxtv.org. Once again its an OpenSource project with outdated confusing documentation, but having a newer Ubuntu Linux kernel really helped the compiles go through smoothly.
All digital broadcasts are transmitted in MPEG-2 transport streams. To record you simply need to dump the MPEG-2 Transport Stream to disk:
mplayer -noconsolecontrols -dumpstream -dumpfile ts-stream dvb://"BBC ONE" & |
To record for a set period of time needs a little shell script. mplayers --end-pos argument does not work as expected.
mplayer -noconsolecontrols -dumpstream -dumpfile ts-stream dvb://"BBC ONE" & PID=$! sleep 60 kill $PID sleep 5 kill -9 $PID |
The TS MPEG-2 file can be played by a number of media players, but I prefer to convert it to DivX:
mencoder -ffourcc DIVX -forceidx -vf scale=704:-2 -ovc xvid -xvidencopts "quant_type=mpeg:bitrate=1500:nopacked" -oac mp3lame -lameopts cbr:br=192 ts-stream -o output.avi |
I have just used a single pass encode with a bit rate of above, but I could use a two pass encode for slightly better quality. Also rather than dumping the MPEG2 Transport Stream to disk it is possible to encode to DivX on the fly. But this needs a fast CPU and the quality would not be as good.
I wrote a simple web PHP front end to schedule and kick off recording:
Looking in the TV guide and manually entering the times into the web GUI feels a bit primitive! I downloaded a TV listings downloader program from xmltv.org. This downloads the listings in XMLTV format. I wrote a perl script to convert the XMLTV to HTML and laid it out with style sheets, so when you click on a program it calls the PHP scripts and adds it to my cron jobs to record.
It now works like any DVR like Sky+. Also because its a web front end on my NASIII Linux Shuttle I can connect to it over SLL from anywhere on the Internet and schedule programs to record. The content is the wirelessly streamed to my PS3 in the lounge to watch.
Memcoder is a multi purpose encoder for Linux and part of the mplayer suite. Since the NAS has the same horse power as my desktop I can offload DVD and other encodes to the NAS. Here's an example of two pass DivX Encoding.
Pass 1. This will give you an indication of bitrate to use.
mencoder -ovc xvid -xvidencopts profile=dxnhtntsc:pass=1:quant_type=mpeg:nopacked input.wmv -oac mp3lame -o /dev/null |
Pass 2. Feed the bitrate parameter into the second pass -xvidencopts bitrate=x:
mencoder -ovc xvid -xvidencopts quant_type=mpeg:pass=2:bitrate=1000:nopacked input.wmv -oac mp3lame -o output.avi |
Before you can re-encode DVDs to DivX on Linux you need to install libdvdcss2.
wget http://www.medibuntu.org/sources.list.d/hardy.list -O /etc/apt/sources.list.d/medibuntu.list apt-get update apt-get install medibuntu-keyring apt-get update apt-get install w32codecs libdvdcss2 |
Use vobcopy -I to identify the main movie. Then use trusty mplayer to pull out the main movie. 1st title in this example.
mplayer dvdnav://1 -dumpstream -dumpfile dump.vob -v |
This dumps the unencrypted VOB to the hard drive. A two pass mencode can then be used to encode it to DivX.
First Pass:
mencoder -ffourcc DIVX -forceidx -vf scale=704:-2 -ovc xvid
-xvidencopts "quant_type=mpeg:pass=1:bitrate=1500:nopacked"
-oac mp3lame -lameopts cbr:br=192 dump.vob -o /dev/null
|
Second Pass:
mencoder -ffourcc DIVX -forceidx -vf scale=704:-2 -ovc xvid
-xvidencopts "quant_type=mpeg:pass=2:bitrate=1500:nopacked"
-oac mp3lame -lameopts cbr:br=192 dump.vob -o dump.avi
|
And now you have a region free backup of your leagally purchased DVD which can now be streamed from my NAS to my PS3.
Mencoder can also produce video files for mobile phones:
mencoder -oac lavc=yes -ovc lavc=yes -of lavf
-lavcopts "acodec=aac:vcodec=mpeg4:abitrate=128:vbitrate=384:autoaspect=yes
:v4mv=yes:trell=yes:mbd=2:vb_strategy=1:aglobal=1:vglobal=1"
-lavfopts "format=mp4:i_certify_that_my_video_stream_does_not_use_b_frames=yes"
-vf scale=320:240 input.avi -o output.mp4
|
Note the cryptic command line. Encoding video files is a total mine field! Steve's Guide to encoding to DivX/Xvid.
There were two problems I faced with email over the years. One was a finding a decent email client. I have been through most of them:
Turnpike --> Outlook Express --> Outlook --> Thunderbird --> Sylpheed.
I moved away from Outlook because Outlooks performance with larger PST files is rubbish. Thunderbird just got too bloated and kept having to reindex mail folders over the network. Sylpheed was a nice fast simple mail client. But this brings me onto the second issue. Downloading all my mail to the NAS, these clients do not let multiple clients access the same central networked mailbox folders.
So looking at archived mail from different computers was problematic. To fix this I installed Dovecot IMAP server on my NAS, and used Getmail to download POP3 mail from my ISP mailbox. Procmail is used to filter the mail into different mail folders as required.
With the server using IMAP I can use any IMAP client to connect to the NAS IMAP server and look at my mail. Unfortunately Slypheed IMAP functionality was pretty poor. This left me looking at other mail clients. I did not want to go back to Outlook or Thunderbird, so I started looking at webmail clients. The advantage of webmail is that I can read all my email including mail archives from any computer with a web browser, including work.
I checked out the following free webmail clients:
SquirrelMail has a simple interface that will work with any browser, but is not really good enough as a main mail client. Atmail is the free version of the commercial program. Its very good but if you have more than a couple of folders the large folder icons mean the folder list goes off screen. Unfortunately there are no skins to sort this out. Roundcube has a very nice simple but powerful interface. So I went with RoundCube.
Using RoundCube and SSL I can now access my email, including mail archives from anywhere in the world.
Next up was an online calendar. Google offers a really good online calendar, but would you put all your personal appointments in the cloud!? Whats to stop Google with their track record indexing and flogging your data to the highest bidder? Instead I installed Webcalendar.
Now I can access my Calendar via SSL anywhere on the Internet. With added email alerts I should never forget peoples birthdays again.
Despite the rest of Theale being wired up for cable, the new build house I am renting has no cable! So I am forced to rent a BT phone line and use ADSL. Being near the exchange I can get 6Mbps peak download speed. But of course even on Unlimited Broadband package with BT Openreach I can get nothing like this with all the bandwidth throttling BT do. I use a Linksys WAG200G modem to connect with.
The really cool feature on this modem is that it can log all packets going through it to a syslog server. In Administration -> Reporting you can specify the IP address of the Log server. I used the NAS.
First I had to enable remote syslogging on the NAS by adding the following line in /etc/defaults/syslogd:
SYSLOGD="-r" |
The Linksys WAG200G logs network traffic as local0, so I configured syslog to log local0 to a separate /var/log/router.log file. And removed local0 from the catch-all logging to messages and syslog.
local0.* /var/log/router.log *.*;auth,authpriv.none;local0.none -/var/log/syslog # # Some `catch-all' logfiles. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none;\ local0.!* -/var/log/messages |
Now you can see ALL packets using your broadband connection, including PS3, everything:
Sep 20 08:46:58 10.0.0.1 UDP Packet - Source:10.0.0.60,50719 Destination:194.74.65.69,53 - [Outgoing] Sep 20 08:47:03 10.0.0.1 UDP Packet - Source:10.0.0.60,43719 Destination:62.6.40.178,53 - [Outgoing] Sep 20 08:47:08 10.0.0.1 UDP Packet - Source:10.0.0.60,40654 Destination:62.6.40.178,53 - [Outgoing] Sep 20 08:47:13 10.0.0.1 UDP Packet - Source:10.0.0.60,37973 Destination:194.74.65.69,53 - [Outgoing] Sep 20 08:47:18 10.0.0.1 UDP Packet - Source:10.0.0.60,37373 Destination:194.74.65.69,53 - [Outgoing] Sep 20 08:47:23 10.0.0.1 UDP Packet - Source:10.0.0.60,58926 Destination:194.74.65.69,53 - [Outgoing] Sep 20 08:47:45 10.0.0.1 UDP Packet - Source:10.0.0.60,52488 Destination:62.6.40.178,53 - [Outgoing] Sep 20 08:47:50 10.0.0.1 UDP Packet - Source:10.0.0.60,58696 Destination:62.6.40.178,53 - [Outgoing] Sep 20 08:47:55 10.0.0.1 TCP Packet - Source:10.0.0.60,47140 Destination:213.123.85.200,80 - [Outgoing] Sep 20 08:48:00 10.0.0.1 UDP Packet - Source:10.0.0.60,60182 Destination:62.6.40.178,53 - [Outgoing] Sep 20 08:48:05 10.0.0.1 UDP Packet - Source:10.0.0.60,56851 Destination:194.74.65.69,53 - [Outgoing] |
This feature has been a long time coming to ADSL modems/routers but I cannot stress how useful a bugging tool this is.
Packing this much punch into a Shuttle is awesome. And this Shuttle is two years old. I just don't need any more horsepower. I do not see the point of Windows Vista that needs more and more powerful computers. Its just marketing hype and bad coding. Just look at the exploding market for low spec Netbooks like the Asus EeePC and Acer Aspire One.
At the time I had concerns over buying a Shuttle rather than an ATX case because it cannot be upgraded. But normally when I want to upgrade it means buying a new motherboard, memory and CPU. And these days everything, sound, ethernet etc, comes on the motherboards. This is now making ATX design redundant. It used to be that you would upgrade component by component using PCI cards. Now this is not the case. I have just built my folks a computer using ATX and there are no PCI cards inside it. Everything is on the motherboard. Its a box full of air!
The question now is how long this Shuttle lasts me. As long as Ubuntu or Gnome does not bloat then there should not be any speed issues. 1Tb drives are selling below 100 pounds but 500Mb is enough for me now. So I will try and extend this Shuttle from my usual 2 year hardware cycle to 4 years. So thats a replacement in 2010 :-)