Saving and restoring bash "shopt" options

Saving and restoring bash "shopt" options

The following command prints out a series of shopt commands to restore your current options.

shopt -p 

as a a series of shopt commands like so

shopt -s xxx
shopt -u yyy
shopt -u zzz

commands.  One minor problem is these commands are newline separated.   

opts=$(shopt -p)
echo $opts     # BAD ack newlines get converted to spaces
echo "$opts"   # GOOD double quotes keep the newlines.

In trying to use $opts, bash does word splitting and turns the newlines into spaces because consecutive newlines, spaces and tabs are converted into a single space.  However enclosing $opts in double quotes preserves the newlines.

someBashFn() {
  local prevOpts=$(shopt -p)

  ... your code which uses shopt ...

  eval "$prevOpts"   # restore the options
}

Formatting large hard drives (2+ TB) in linux

There are two steps

• create a partition using parted  (the key is to use the -a optimized flag)
• make the filesystem using mke2fs (or mke4fs or mkfs.e4fs)

The venerable fdisk does not handle disks larger than 2TB so you must use parted.

Parted

Here's what we will do

1. find the name of the disk you want to format with fdisk
2. use parted to 
• create a partition table (which is called a "label" in parted, ugh) of type gpt 
• make the partition (which only reserves space on the disk)
• (optional) name the partition in the partition table
3. format the partition 
• show options to get the most space out of the partition

Here are the commands:

  $ sudo bash       # become root, avoid all the sudo prefixing

  # fdisk -l        # find the disk you want to partition

     ... find the disk you want to partition 

     ... if it is new, there will be no partitions associated with it

     ... lets assume the desired disk is /dev/sdY


// Next, It is important to use -a optimized, other wise you will get the dreaded
// Warning: The resulting partition is not properly aligned for best performance.
// You can try to calculate how to align the partitions but it is 10 minutes of digging around 
// disk specs and worse, none of the advice worked from various web sites (and here).
// Instead let parted do the work.
// Printing the start and end of the disk showed an offset remains a mystery to me,

  # parted -a optimized /dev/sdY

    // Create the partition table of type gpt

 (parted) mklabel gpt

    // Create a primary partition from the first sector to the last.

    // It is important to use 1 not 0 as the first partition if you want "optimal alignment"

 (parted) part primary 1 -1

  // The next step is optional - give your partition a name.  The name is solely for

    // your convenience.  It is only visible in parted (and possiblly fdisk)

  (parted) name 1 your-name-of-your-partition

  (parted) print

  (parted) quit

mke2fs or mke4fs

The default parameters to mke2fs are very outdated for 

1. reserving emergency administrator space (the -m flag) and 
2. the bookkeeping for files (the -i flag which is the avg file size for each inode)

These values assumes files are small text and disks are smaller than 10GB (!).

With a 2+TB disk, 1% space for admin is enough (20 GB) and the average file size is at least 16KB (think of all the photos, videos, music and PDFs).

    # mkfs.ext4 -m 1 -i 16384 dev/sdY

If you want to squeeze very last bit of space out of your disk and you mostly have large files, say averaging 64KB) and you do not use extended file attributes then you can use

    # mkfs.ext4 -m 0 -i 65536 -I 128 dev/sdY

Details: In modern linux kernels the default inode is 256 bytes.  The file system needs one inode per file and the inodes are reserved when you format the partition.  If you run out of inodes, you cannot create any new files (the disk will appear to be "full").  However, inodes use space.  E.g. if you use the default of an inode (256B) for every 4K of data, then 1/17 ~ 6% of your space is used for inodes.  On a 4TB disk, this is enough inodes for 1 billion files.

If we reserve an 256-byte inode for every 64K then only 256/(256+64K) = 1/257 = 0.5% of disk is spent on inodes.

Finally, the -I flag indicates to use the old inode size of 128, in which case you lose the ability to set extended Linux file attributes which are rarely used, thus reducing the inode overhead by half again.  

Mac OS X - fix boot problem due to hard disk errors

Returning from a 2 week trip to the Singapore and Myanmar (aka Burma), my Mac Book Pro 13 which had sat at home powered off the whole time, refused to boot.  About 15% throught the progress bar when booting, the Mac would power off.

Looking up keyboard shortcuts, I held down the 'd' key to show diagnostics, which indicated the mac was failing during the automatic fsck (disk drive file system check).

The solution:

1. Boot to single user, by holding the 's' key once the startup gong sounds.
2. Try /sbin/fsck -fy which forces a fix answering "yes" to all questions.  Repeat this command until no errors are reported.
3. In my case, the "catalog" tree structure was broken enough that fsck stopped without fixing the problem.
4. I ran fsck_hfs directly which has additional options not supported by the generic fsck, shown by man fsck_hfs.  In the following command, I used the raw device (/dev/rdisk0s2, for my laptop, yours maybe different) for my hard drive that the previous fsck had shown in step 2:
      /sbin/fsck_hfs -Rc /dev/rdisk0s2After this I reran /sbin/fsck -fy to verify the disk had been reconstructed.
5. Restart the computer by exiting out of single user mode:  exit

Linux: Samba starts before eth0 is up and will not serve... ack.

My main SMB server, call it sammy, was upgraded to Linux Mint 13 system, but it was not serving.

I noticed that after I started the system (sammy), but before I had logged in, I could not ssh into sammy.  After logging in to sammy on the console, I was able to ssh into it.  It was a surprise to not find it from other SMB clients.

After much debugging, the main clue came when I was on the SMB server, sammy.  Connecting via localhost loopback IP address, I see the SMB service fine when I ran
  smbclient -L localhost -N
but when I tried to connect via the ether IP address, nothing showed up 
  smbclient -L sammy -N   I

The long and short of it was samba was starting before the ethernet interface was getting assigned, so the smb server was unaware of that interface.  Restarting samba solved the problem via
   service smbd restart
   service nmbd restart

Transfer files off your Android phone: use Droid NAS as an SMB server with a Mac or Linux

If you are getting a getting rid of an Android phone  or device, it's best to make a full backup before you factory reset it.   While, there are many ways to transfer files off of your Android device, I wanted to get everything by viewing my Android files from another "client" computer (which is where I'll copy the Android data).  The steps are:

1. Export the Android files via the SMB/ CIFS protocol (which is how Windows shares files across computers)
2. Mount the exported Android  "shares" (the root folders/directories) on a client computer, which can be either Mac or Linux, so you can see the Android files.  (It has been reported that windows will not work as Droid NAS exports via a non standard port; I have not verified if there is a workaround.)  Note: to use Linux you need to be able to run programs as root via sudo.
3. Copy the files on the client from the Android device to your client computer, using rsync.

(1) Export or expose your Android data to another computer.
There are many Android programs that run as an SMB server.  The one I used is Droid NAS.   After starting the server in Droid NAS, it shows the IP of the Android device and a port, for example 7777, which is the value I will use in the following examples.  There are three profiles "Home", "Work", and "Cafe".  I chose "Home".  I also went to Settings (the gear icon on the bottom left) and specified the wi-fi network on which to export, and I added a user and password, which I'll assume are "uuu" and "pppp" in the examples.  To get these new settings applied, I stopped the server and restarted it.

(2) To see the Android files do either of the following:
On a Mac, in the finder, you should see the Android device in the Shared Section.
Since we added a user and password, you'll have to connect as user uuu with your password pppp.  The finder should show several "shares" that you can "connect" or "mount" on your Mac.

On my phone, I had four shares: Camera, Downloads, Photos and SD Card.  You may have more or fewer shares.

On a linux machine: Verify your linux kernel supports cifs.  You should see a line with cifs when you run
  % grep cifs /proc/filesystems
If you don't see anything, try using smbfs instead of cifs.  If you still don't see anything, you need a newer kernel.  Stop now.

(Optionally, though I never got this to work) verify you can see the Droid NAS as an SMB server:
 % smbclient -L Android-IP-address -p port -U uuu

Mount a share on the linux client.  First make a directory on the Linux cilent where the Android files will be mounted, say /mnt/Android/Camera.
  % sudo mkdir -p /mnt/Android/Camera

 Mount the Android share via the following command.
  % sudo mount -o ro,port=7777 -t cifs -o username=uuu,password=ppp //Android-IP-address/Camera /mnt/Android/Camera

(3)  Copy the files to your client computer.
On the mac:
  (a)  drag the Android shares where you want to copy them.  If this does not work, perhaps because you have too many files and the finder seems to time out getting all the files to copy
  (b) Open a terminal on your Mac and type
   % df
  See where the Android shares are mounted (look in the last column), e.g. /Volumes/Camera
  Copy each share them with rsync via
  % rsync -axv /Volumes/Camera /folder/holding/the/backup

On the linux machine, run rsync
  % rsync -axv /mnt/Android/Camera /folder/holding/the/backup

Upgrading your TiVo HD hard drive with a bigger HD

TL;DR:

For Tivo HD (652160) and Series 3.   Get the command via the interactive command generator at MFS.  Download the boot disk from 2009, from the release announcement (see the attachment).  

If you are putting in a 1-2TB HD the command should look like:
       backup -qTao  -  /dev/sd-original | restore -s 500  -xzpi  -   /dev/sd-upgrade

Run fdisk -l to determine your hard drives.  Replace /dev/sd-original with your original drive, which will be something like /dev/sda, and also for /dev/sd-upgrade.

The -s 500 indicates to use 500MB as swap, which is larger than the default 127.  (Failed: I though I could use "restore -r 8 ... " to indicate the block size in MB for recordings, but the value has to be between 1 and 4.   With HD recordings being 5+GB, I thought using -r 8 or even -r 16 would help free up memory and would cause minimal internal fragmentation.)

The maximum size you can upgrade to is 2TB as of 2013.  Choose a HD with low noise and low power consumption (heat); typically you want an "Eco" or "green" drive that has a slow RPM, say 5400 or even 4200.  This is what Tivo uses.  All HDs can record and play back streaming material at 10X the speed needed.  Try to avoid a 7200 RPM drive if possible.

For a longer explanation on the TiVo HD / Premiere: read Ross Walker's detailed blog or for the new Premiere TiVos see the community discussion with links to the newest software.

For newer Tivos, see the community posting and also this newer update about using the JMFS tools. -------------------------- 
 Ignore, since this is all said better and with pictures at Ross Walker's blog.
Details:

Tivo lets you upgrade from the factory-installed hard drive to your own hard drive.    Why do this?  It's much much cheaper.  And you get a much bigger hard drive, such as 2T.


There is a lot of old information on the web, that evolved as Tivo introduced various models.  But since the switch to digital, aka HD, TV, you need a "newer" TiVo that has a digital tuner.
We got a bunch of the HD (which are the same as the Series 3).  Since then, Tivo has come out with a "Premiere" line, which this blog post does not cover.

Here's what to do to upgrade your Tivo HD or Series 3:

0) Need: computer, large SATA drive, Torx-10 and Torx-15 screwdrivers, blank CD, CD burner.

1) Find a computer that can boot from CD, that you can plug 2 SATA drives into.  It's OK to put these drives into external USB enclosures and hook them up via USB.  I have only done this hooking the HDs to a M/B directly via the SATA connectors.

Download the tools from MFS.  This is a tech blog, get the linux distro and burn to a CD.

2) Get a Torx 10 and Torx 15 screwdriver.  Open your Tivo box, via the 6 torx-10 screws on the back.  Slide the top case toward you by 1/4 to 1/2" and pull the sides outward gently and wiggle the case off.

3) Remove the original HD from the TiVo.:  Remove the SATA / power cable from the HD.
The HD is fastened by via 4 Torx-10 screws. Pull the HD + metal holder out.  Remove the metal stand held to the HD by 4 Torx-15 screws.

4) Hook up the original drive and the new drive to your computer, and boot the computer with your MFS CD.

5) Finish this.

Keebox W150NR V2.0R routers are not the DLink DIR-300vB

TL;DR: this Keebox is not the same as a DLink DIR-300vB.  It is not DD-WRT compatible as of Feb 2013.

----------------------------------------------------
Fry's was selling Keebox W150NR routers for very little ($10 or $15) in late 2012.  I got two.

My knowledgable friend Google told me these were rebranded DLink DIR-300vB routers.  The important things to know about these routers are

1. The version matters, as version A is completely different hardware than version B.  The Keebox is a DIR-300vB.
2. There are some pretty bad security vulnerabilities with some (all?) of this model of DLink routers that will not be fixed.  
3. Some people managed to install DD-WRT on their Keebox routers by using the DIR-300vB instructions with minor modifications, but I cannot, since the routers I got behave differently than the others.  Sigh.

My Keebox router

My routers are labelled: HW: V2.0R FW:2.002 (note the 'R' in the version)


My Keebox behaves differently than the DIR-300vB and also differently from the other Keebox routers mentioned in this DD-WRT post.

1. On normal power up, it has IP 192.168.10.1 as others have mentioned.
2. When booting in the "Emergency Room Web Interface", the router comes up with IP 192.168.123.254 (not 192.168.10.1).  To access the upload page, you must let the router assign an IP address to your computer via DHCP.  Then you can visit http://192.168.123.254   
3. However, none of the DD-WRT images would upload correctly.  I tried uploading
   from every variant of browser on both Win and Mac.
4. The factory firmware does not have the security vulnerabilities of DLink-300vB , e.g. /command.php is not open.

It turns out this Keebox V2.0R is not the same as a DLink DIR-300vB.  I opened the router, by removing the two black pads on the bottom, and unscrewing the two screws underneath.  The SoC is a Ralink RT5350F (not a RT3050F) and  there is a chip missing relative to the reference photo.  After I figured this out, a better Google search corroborated my findings.

The Keebox summary: version 1 is good, but version 2 is not good.   Thus HW: V1.0R is a DLink DIR-300vB, but HW V2.0R is possibly a DIR-600v5 which does not support DD-WRT as of 2/2013.