Lyle Backenroth » Command Line

Command Line Magic: Part III

lyle — Sat, 20 Mar 2010 08:55:09 +0000

As part of my continuing Command Line Magic series and many of the other Command Line oriented posts I’ve made (click here for category-summary of Command Line oriented posts, or just click the Command Line tag in the tag cloud to the right), I’m happy to post another set of highly useful commands. As always, the context of these commands are within the Bash shell in Linux. A moderate understanding of Bash shell commands is required to fully appreciate this post.

Here are some very useful commands, that any power user would find helpful:

1. Start a simple webserver to serve up any directory as browsable from anywhere (for file transfers):

$ python -m SimpleHTTPServer

I’ve mentioned this in past posts. This is a simple command, that when run from any directory will launch a simple python web server that will serve up the local directory as a browsable directory using a browser such as Firefox or Chrome. Any subdirectories underneath the local directory from which this command is run will also be browsable. You can right-click and save any file or left-click it to attempt to view it on the fly. This works very well over SSH sessions, when you want to transfer a file, but don’t want to engage SSHFS or SCP. You can background the process with a ctrl-z, bg, then pkill python to stop the web server from running, or just leave it running in command prompt and ctrl-c to end it.

2. Record your desktop and pipe the output to an mpeg file.

$ ffmpeg -f x11grab -s wsxga -r 25 -i :0.0 -sameq /home/john/desktop.mpg

-f allows ffmpeg to grab the data properly from the x11 framebuffer
-s sets the size of the screen to actually record, starting from the upper left of the screen. Here wsxga denotes a specific preset resolution (in wsxga’s case that would be 1600 x 1024). You can however type any resolution you like in manually (e.g. -s 1024×768). You will need to know the resolution of your desktop to set this correctly.
-r sets the framerate. This could be left out as 25 is the default.
-i sets which framebuffer to take, since XWindows can run in multiple sessions, generally you’ll want to leave this setting alone.
-sameq forces the same quality was what is being fed in by the source (in this case the x11 framebuffer). This is helpful to have a max-quality video, though you may want to try other settings to degrade the quality to keep the file size down. If you’d prefer to reduce the quality on the fly, replace -sameq with -qscale x where x is 1 – 31. These are preset quality settings, with 1 being the highest and 31 being very poor video quality. I have found -qscale 10 to be the sweetspot between quality and file size.
If you’d like the file to be a bit smaller and if you prefer an .AVI to a raw .MPG, then simply remove the /home/john/desktop.mpg in the command above and replace it with:
- -vcodec mpeg4 /home/john/desktop.avi
  - This is file will be a bit smaller using the mpeg4 codec in an avi container. You can still use the -qscale option with this change.

3. Copy an entire directory tree through ssh using on the fly compression through an SSH session (no temporary files!):

$ ssh  'tar -cz //' | tar -xvz

Just enter the to SSH to, and the host’s and path to compress that subdirectory on the fly at the host, but decompress it as it arrives locally to your current location and path. This will have the advantage of not taking up any extra space at the host (since the files are compressed as they’re transmitted) and easily drops the entire directory tree specified onto the client uncompressed, saving time and bandwidth and transmission time.

This works well for large directory trees and is easy to use for a quick copy where you don’t want to spend a lot of time compressing it at the host manually and transmitting the compressed file, then uncompressing it, then deleting the original compressed file created at the host. Note: This will replicate the full directory path at the client side (desired).

SCP or RSYNC are recommended for automated backup though, this is more appropriate for a 1-shot copy of a large directory.

4. Resize any image files in the current directory to Width x Height specifed (regardless of image format)!

$ for a in `ls`; do echo $a && convert $a -resize x $a; done

Simply do a man convert to learn more about the convert program, other options can be added into the command. Also this is a great syntax for doing ANYTHING to any files in a particular directory that would be a batch process consistent with all the files in that directory.

5. Grab a screenshot of the current desktop to the current directory

$ import -pause 5 -window root desktop_screenshot.jpg

This command will wait 5 seconds (assuming you want some time to set up the shot and to get the command prompt out of the way) and take a snapshot of the root (primary) desktop currently running. This command requires imagemagick be installed.

Screen Profiles available Ubuntu 9.04

lyle — Tue, 28 Apr 2009 17:58:06 +0000

Interesting article about .screenrc profiles that offer some unique status bars with GNU Screen under Ubuntu 9.04

For any that are interested, this is my .screenrc

hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%Y-%m-%d %{W}%c %{g}]'
#
# Default screens
# syntax: screen -t NameOfScreen ScreenNumber ShellCommand
screen -t "Work SSH" 0 /home/name/workssh
screen -t "Home 1" 1
screen -t "Home 2" 2
screen -t "Home 3" 3
screen -t "Home 4" 4

The hardstatus string gives me the local computer’s hostname on the left, lists the shells themselves in the middle, and on the right offers the date and time.

Line 6 with “Work SSH” executes the shell script “workssh”, labels the window “Work SSH” and assigns it to virtual shell number 0. This shell script allows me to SSH to one of my work servers offering me a series of local listening ports that forward to remote IP’s and ports.

Shells 1-4 are local bash shells.

Linux Reference Sheets

lyle — Fri, 29 Aug 2008 06:21:06 +0000

The 1 page Linux CommandManual

Linux Command Reference

Linux Security Quick Reference

The Gimp Hotkey Reference

Command Line Magic: Part II

lyle — Sun, 27 Jul 2008 21:14:03 +0000

Fun with rename:

1) To rename filenames with CAPS characters to lowercase:

rename ‘y/A-Z/a-z/’ *

2) To remove from filenames:

rename ’s/ /-/g’ *

3) To strip files with the .bak extension:

rename ’s/\.bak$//’ *.bak

Free: The free command displays the amount of free and used memory in the system. For example, free -m gives the information using megabytes, which is probably most useful for current computers.

lsb_release -a: The lsb_release command with the -a option prints version information for the Linux release you’re running

sed: The sed (or Stream EDitor) command allows search and replace of a particular string in a file. For example, if you want to find the string “cat” and replace it with “dog” in a file named pets, type
sed s/cat/dog/g pets.

grep: An excellent tutorial.

Running commands sequentially in one command:
If you need to execute multiple commands in sequence but don’t need to pass output between them, there are two options based on whether or not you want the subsequent commands to run only if the previous commands succeed or not. If you want the commands to run one after the other regardless of whether or not preceding commands succeed, place a ; between the commands.

For example, if you want to get information about your hardware, you could run lspci ; lsusb which would output information on your PCI buses and USB devices in sequence.

However, if you need to conditionally run the commands based on whether the previous command has succeeded, insert && between commands. An example of this is building a program from source, which is traditionally done with ./configure, make, and make install.

The commands make and make install require that the previous commands have completed successfully, so you would use ./configure && make && sudo make install. Usually you’ll need an SUDO, for the last command, so make sure you’ve authenticated SUDO prior to running that command.

Learning the Linux Shell: A great tutorial.

10 Essential tips of Linux Admins

lyle — Sun, 27 Jul 2008 06:24:21 +0000

Tips from IBM, not all that complicated, but worth looking at.

Other tips include…

Sharing computers on a Linux (or heterogeneous) network.
IBM’s top 10 Linux articles and tutorials.
Miscellaneous tips.
Linux-powered networking: Part 1, Part 2 and Part 3.
Industrial-strength Linux box lockdown, a howto: Part 1 and Part 2.
Introduction to cryptology concepts: Part 1, Part 2 and Part 3.

Note: Some of these links requires registration.

Downloading torrents from command line

lyle — Sun, 27 Jul 2008 06:11:55 +0000

Source: LinuxHaxor.net

Essentially RTorrent, CTorrent and Transmission-CLI.

CLI Magic

lyle — Thu, 24 Jul 2008 00:13:38 +0000

Source for this great article: Linux.com

In this context, a one-liner is a set of commands normally joined through a pipe (|). When joined by a pipe, the command on the left passes its output to the command on the right. Simple or complex, you can get useful results from a single line at the bash command prompt.

For example, suppose you want to know how many files are in the current directory. You can run:

ls | wc -l

That’s a very simple example — you can get more elaborate. Suppose you want to know about the five processes that are consuming the most CPU time on your system:

ps -eo user,pcpu,pid,cmd | sort -r -k2 | head -6

The ps command’s o lets you specify the columns that you want to be shown. sort -r does a reverse order sort with the second column (pcpu) as reference (k2). head gets only the first six lines from the ordered list, which includes the header line. You can place pcpu as the first column and then omit the k2 option because sort by default takes the first column to do the sort. That illustrates how you may have to try several approaches on some one-liners; different versions and ways to manipulate the options may produce different results. A common situation for Linux administrators on servers with several users is to get quick ordered user lists. One simple way to get that is with the command:

cat /etc/passwd | sort

If you just need the username, the above command returns too much information. You can fix it with something like this:

cat /etc/passwd | sort | cut -d":" -f1

The sorted list is passed to cut, where the d option indicates the field’s delimiter character. cut breaks into pieces each line, and the first field f1 is the one that you need to display. That’s better; it shows only usernames now. But you may not want to see all the system usernames, like apache, bin, and lp. If you just want human users, try this:

cat /etc/passwd | sort | gawk '$3 >= 500 {print $1 }' FS=":"

gawk evaluates each line from the output piped to it. If the third field — the UID — is equal or greater than 500 (most modern distros start numbering normal users from this number) then the action is done. The action, indicated between braces, is to print the first field, which is the username. The separator for field in the gawk command is a colon, as specified by the FS option. Now suppose you have a directory with lots of files with different extensions, and you want to back up only the .php files, calling them filename.bkp. The next one-liner should do the job:

for f in *.php; do cp $f $f.bkp; done

This command loops through all the files in the current directory looking for those with .php extensions. Each file’s name is held in the $f variable. A simple copy command then does the backup. Notice that in this example we used a semicolon to execute the commands one after another, rather than piping output between them. What about bulk copy? Consider this:

tar cf - . | (cd /usr/backups/; tar xfp -)

It creates a tar package recursevely on the current directory, then pipes this package to the next command. The parenthesis creates a temporary subshell, changes to a different directory, then extracts the content of the package, which is the whole original directory. The p option on the last tar command preserves file properties like time and permissions. After completion, the shell context will be at the original directory. A variant on the previous one-liner lets you do the same kind of backup on a remote server:

tar cf - . | ssh smith@remote.server tar xfp - -C /usr/backup/smith

Here, the command establishes an SSH remote session and untars the package with the C option, which changes the directory, in this case to /usr/backup/smith, where the extraction will be made.

grep and gawk and uniq, oh my!

Text processing is a common use for one-liners. You can accomplish marvelous things with the right set of commands. In the next example, suppose you want a report on incoming email messages that look like this:


cat incoming_emails
2008-07-01 08:23:17 user1@example.com
2008-07-01 08:25:20 user2@someplace.com
2008-07-01 08:32:41 somebody@server.net
2008-07-01 08:35:03 spam not recived, filtered
2008-07-01 08:39:57 user1@example.com
...

You are asked for a report with an ordered list of who received incoming messages. Many recipients would be repeated in the output of the cat command. This one-liner resolves the problem:

grep '@' incoming_email | gawk '{print $3}' | sort | uniq

grep filters the lines that contains a @ character, which indicates an email address. Next, gawk extracts the third field, which contains the email address, and passes it to the sort command. Sorting is needed to group the same recipients together because the last command, uniq, omits repeated lines from the sorted list. The output is shown below. Most text processing one-liners use a combination of grep, sed, awk, order, tr, cut, uniq, and other related commands.


somebody@server.net
user1@example.com
user2@someplace.com

If you like any of these one-liners but think they’re too long to type often, you can create an alias for the command and put it in your .bashrc file. When you log in your session, anything inside this file will be run, so your personal aliases would be ready at anytime.

alias p5="ps -eo pcpu,user,pid,cmd | sort -r | head -6"

You can certainly create better and simpler variations of all of the commands in this article, but they’re a good place to start. If you are a Linux system administrator, it’s good practice to collect, create, and modify your own one-liners and keep them handy; you never know when are you going to need them. If you have a good one-liner, feel free to share it with other readers in a comment below.

Useful less-used Linux commands

lyle — Sun, 20 Jul 2008 22:15:59 +0000

1. lsb_release -a

2. The watch command.

3. Command line intro summary, with some rarely used switches (TLDP).

4. If you’re using a system which has a lot of users, and you’d like to see who has started a particular script, daemon, or binary, then the pstree utility is very helpful. It draws a tree of all currently running processes – allowing you to see which processes are related.