Saving and restoring bash "shopt" options

Saving and restoring bash "shopt" options

The following command prints out a series of shopt commands to restore your current options.

shopt -p 

as a a series of shopt commands like so

shopt -s xxx
shopt -u yyy
shopt -u zzz

commands.  One minor problem is these commands are newline separated.   

opts=$(shopt -p)
echo $opts     # BAD ack newlines get converted to spaces
echo "$opts"   # GOOD double quotes keep the newlines.

In trying to use $opts, bash does word splitting and turns the newlines into spaces because consecutive newlines, spaces and tabs are converted into a single space.  However enclosing $opts in double quotes preserves the newlines.

someBashFn() {
  local prevOpts=$(shopt -p)

  ... your code which uses shopt ...

  eval "$prevOpts"   # restore the options
}

Formatting large hard drives (2+ TB) in linux

There are two steps

• create a partition using parted  (the key is to use the -a optimized flag)
• make the filesystem using mke2fs (or mke4fs or mkfs.e4fs)

The venerable fdisk does not handle disks larger than 2TB so you must use parted.

Parted

Here's what we will do

1. find the name of the disk you want to format with fdisk
2. use parted to 
• create a partition table (which is called a "label" in parted, ugh) of type gpt 
• make the partition (which only reserves space on the disk)
• (optional) name the partition in the partition table
3. format the partition 
• show options to get the most space out of the partition

Here are the commands:

  $ sudo bash       # become root, avoid all the sudo prefixing

  # fdisk -l        # find the disk you want to partition

     ... find the disk you want to partition 

     ... if it is new, there will be no partitions associated with it

     ... lets assume the desired disk is /dev/sdY


// Next, It is important to use -a optimized, other wise you will get the dreaded
// Warning: The resulting partition is not properly aligned for best performance.
// You can try to calculate how to align the partitions but it is 10 minutes of digging around 
// disk specs and worse, none of the advice worked from various web sites (and here).
// Instead let parted do the work.
// Printing the start and end of the disk showed an offset remains a mystery to me,

  # parted -a optimized /dev/sdY

    // Create the partition table of type gpt

 (parted) mklabel gpt

    // Create a primary partition from the first sector to the last.

    // It is important to use 1 not 0 as the first partition if you want "optimal alignment"

 (parted) part primary 1 -1

  // The next step is optional - give your partition a name.  The name is solely for

    // your convenience.  It is only visible in parted (and possiblly fdisk)

  (parted) name 1 your-name-of-your-partition

  (parted) print

  (parted) quit

mke2fs or mke4fs

The default parameters to mke2fs are very outdated for 

1. reserving emergency administrator space (the -m flag) and 
2. the bookkeeping for files (the -i flag which is the avg file size for each inode)

These values assumes files are small text and disks are smaller than 10GB (!).

With a 2+TB disk, 1% space for admin is enough (20 GB) and the average file size is at least 16KB (think of all the photos, videos, music and PDFs).

    # mkfs.ext4 -m 1 -i 16384 dev/sdY

If you want to squeeze very last bit of space out of your disk and you mostly have large files, say averaging 64KB) and you do not use extended file attributes then you can use

    # mkfs.ext4 -m 0 -i 65536 -I 128 dev/sdY

Details: In modern linux kernels the default inode is 256 bytes.  The file system needs one inode per file and the inodes are reserved when you format the partition.  If you run out of inodes, you cannot create any new files (the disk will appear to be "full").  However, inodes use space.  E.g. if you use the default of an inode (256B) for every 4K of data, then 1/17 ~ 6% of your space is used for inodes.  On a 4TB disk, this is enough inodes for 1 billion files.

If we reserve an 256-byte inode for every 64K then only 256/(256+64K) = 1/257 = 0.5% of disk is spent on inodes.

Finally, the -I flag indicates to use the old inode size of 128, in which case you lose the ability to set extended Linux file attributes which are rarely used, thus reducing the inode overhead by half again.