The terminal emulator that ships with macOS is, by default, a black (background) and white (text) affair. In addition to a bland aesthetic, this lack of color has practical implications for the user. For example, the output of the ls command is muted, while files opened in vim lack syntax highlighting.

For those that want to add a little color to their terminal, brightening things up is only a few keystrokes away. First, if it does not already exist, create a .bash_profile in your home directory with: touch ~/.bash_profile, then edit this file with: vim ~/.bash_profile to include the following:

export CLICOLOR=1
export LSCOLORS=GxFxCxDxBxegedabagaced

Next, open up a terminal and push these changes with: source ~/.bash_profile. Lastly, check "Display ANSI colors" in Terminal -> Preferences -> Profiles -> Text (if needed).

Similarly, adding color to the vim palette involves creating a .vimrc file in your home directory with: touch ~/.vimrc, and then editing the file with vim ~/.vimrc to include:

syntax on

These (colorful) changes take effect once you save your edits.

Cheers.

Sed--the stream editor--can read, then modify text on the fly. While the syntax for sed appears cryptic at first blush, it is a versatile text-processing tool worth adding to your UNIX utility belt.

In the real world, one could use sed in combination with the find utility to recursively locate, then remove: 1) all HTML elements, and 2) all leading whitespace from the .php files of a website.

Here's one approach to that task:

#!/usr/bin/env bash

find ./ -type f -name '*.php' \
  -exec sed -i 's/<[^>]*>//g; s/^[ \t]*//g ' {} \;

Let's take a look at the find portion of the snippet first. For reference, the syntax for this command is:

find [location to search] [expression] [-option(s)] [what to find]

In our example, we tell find to begin from the present working directory ./, and search only for files -type f that end with the '*.php' extension. We then instruct find to -execute two (2) sed commands.

Ouch! Those sed commands look evil. We'll break them down, but before we do, pay attention to the syntax for the command:

sed Options... [script] [inputfile]

and note the following:

-i = in-place edit
s = substitute
g = global
// = delete

Our first expression:

s/<[^>]*>//g

• looks for an opening tag <
• which is followed by zero or more characters *, which are NOT ^ a closing tag >
• then looks for a closing tag >

replacing matches with nothing.

Our second expression:

s/^[ \t]*//g

• looks for lines that start ^ with tabs \t

replacing matches with nothing.

Find then brings it home by recursively {} running our commands, and draws to a close with the semi-colon ;.

Viola! With a little elbow grease, we were able to transform reams of text in to a format suitable for export elsewhere.

Cheers.

At its core, the command line interface (CLI) grants users control over complex systems, and allows for the automation of tasks that would otherwise be cumbersome to manage manually. What follows is a brief transcript that helps to illustrate the point.

last | awk '{print $3}' |grep -E '128.122..'|sort -V |uniq >> publicIPs

Here, from our host node, we get a list of logins via the last command, extracting only the third (3rd) column of the output via awk, then grep for IPs on the 128.122 subnet. From there, we sort said IPs numerically, and output uniq IPs to a list ("publicIPs"). For reference:

128.122.87.178
128.122.113.43
128.122.113.212
128.122.114.69
128.122.114.139
128.122.115.60

Next, let's iterate through this list (by line), performing an nslookup on each IP, and parse our region of interest (ROI),then add this ROI to a file ("dnsRecs").

while IFS= read -r line; do
  nslookup "$line" |awk 'FNR == 1 {print $4}' >> dnsRecs
done < publicIPs

By way of example (FQDN omitted):

plabrigla
publio
tamino
figaro
keystone
syndrome

Then, let's iterate through the list of DNS records, using nmap to gather some intelligence on the hosts:

while IFS= read -r line; do
  echo "$line" >> osRecs
  nmap -O "$line" >> osRecs
done < dnsRecs

For reference (single host, DNS ommitted):

Not shown: 996 closed ports
PORT      STATE    SERVICE
22/tcp    filtered ssh
3283/tcp  open     netassistant
5900/tcp  open     vnc
28201/tcp open     unknown
Aggressive OS guesses: VMware ESXi 3.5 (91%), FreeBSD 6.1-RELEASE (88%),   FreeBSD 6.2-RELEASE (88%), Apple iOS 4.3.1 - 4.3.5 (87%), 
Apple iOS 6 (87%), Apple iOS 8.2 (Darwin 14.0.0) (87%), Apple OS X 10.10 (Yosemite) or iOS 8.3 - 9.0.1 (Darwin 14.0.0 - 15.0.0) (87
%), Apple iPhone OS 3.1.2 - iOS 4.2.1 (87%), Apple Mac OS X 10.5 (Leopard) (Darwin 9.2.2, x86) (87%), Apple Mac OS X 10.5.5 (Leopar
d) - 10.6.6 (Snow Leopard) (Darwin 9.5.0 - 10.6.0) (87%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 8 hops

Cheers.

Here's a quick-and-dirty bash snippet that recursively locates all files ending with the .npy extension, and then replaces occurrences of a colon(:) with a dash (-) in their filename . For reference:

find ./ -depth -type f -name '*.npy' \
    -exec bash -c 'mv "$0" "${0//:/-}"' {} \;

If you'd like, you could run the following to create a sandbox for testing:

#!/usr/bin/env bash
# Create time stamp

timestamp=$(date +"%H:%M:%S")

# Create three (3) dummy directories with two-hundred (200) files in each. A file will take this form: 
dummy-data-000:20:12:00.npy

mkdir -pv data{0..2} && touch data{0..2}/dummy-data-{0..1}{0..9}{0..9}:"$timestamp".npy

Cheers.

Speed Test by Ookla is a web service that
measures Internet connection performance metrics like download/upload rate, latency, and packet loss. The company offers a browser-based test, as well as several dedicated applications for the same, though you can bypass all of that with this snippet:

wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip

Here we ask wget to download a 100 MB zip file from Speed Test, and send the result to /dev/null, where it's immediately discarded. The results are conveyed in real-time to your terminal, e.g.:

--2021-11-21 10:40:00--  http://speedtest.wdc01.softlayer.com/downloads/test100.zip
Resolving speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)... 169.54.48.218, 2607:f0d0:3006:154::3
Connecting to speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)|169.54.48.218|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104874307 (100M) [application/zip]
Saving to: ‘/dev/null’

100%[====================================================================================================>] 104,874,307 72.5MB/s   in 1.4s 

2021-11-21 10:40:01 (72.5 MB/s) - ‘/dev/null’ saved [104874307/104874307]

Keep in mind that this command-line test only measures download speed, and the result is limited by the lower of the maximum speed the source/destination can offer. Also, it's useful to poll multiple sources, and run multiple tests. To perform the latter, you can do:

for run in {1..5}; wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip; done

which runs the test five (5) times sequentially.

For more information on Speed Test results, see here. Also, check out the command line application here.

Cheers.

In our last post we discussed using Bash to transform data in one format to an arrangement suitable for import in to Google Groups. Now, let's take a soup-to-nuts look at the process.

For reference, our source data takes the following format:

# list-o-Steves
#
Stephen Gary Wozniak <imalegend@example.com>
Stephen William Hawking <imaboygenius@example.com>
Steven Anthony Ballmer <imapc@example.com>
Steve Austin <imastonecoldsunofagun@example.com>
Steven Paul Jobs <imamac@example.com>
Stevland Hardaway Morris <imawonder@example.com>.

and each individual record in the source needs to be modified to the following:

,<email@example.com>,USER,MEMBER,Firstname.

So how can we get there? Well, we can start by discarding lines that begin with a # in our source file, as they are irrelevant. To do that, tell grep to not (^) print lines that begin with #:

grep "^[^#]" "$file".

This yields:

Stephen Gary Wozniak <imalegend@example.com>
Stephen William Hawking <imaboygenius@example.com>
Steven Anthony Ballmer <imapc@example.com>
Steve Austin <imastonecoldsunofagun@example.com>
Steven Paul Jobs <imamac@example.com>
Stevland Hardaway Morris <imawonder@example.com>.

Notice that in each row of the returned list, a column is separated by one (1) blank space. That's awfully useful for our purposes, as we can inform awk that a blank space " " indicates a separate field (-F " "). Then, when awk reads through each row, we can tell it exactly what we want to print:

'{print "", $NF, "USER", "MEMBER", $1}',

which yields:

,<imalegend@example.com>,USER,MEMBER,Stephen
,<imaboygenius@example.com>,USER,MEMBER,Stephen
,<imapc@example.com>,USER,MEMBER,Steven
,<imastonecoldsunofagun@example.com>,USER,MEMBER,Steve
,<imamac@example.com>,USER,MEMBER,Steven
,<imawonder@example.com>,USER,MEMBER,Stevland

That's nifty! Here's a breakdown:

Taken all together:

grep "^[^#]" $file | awk -v OFS=, -F " " '{print "", $NF, "USER", "MEMBER", $1}',

wherein we pipe (|) the results from grep to awk, and tell awk to separate each value (-v OFC=,) it prints with a comma (,). We can then extend the example by redirecting standard output to the .csv file we created in Part 1 of this project with:

>> "$file"."$file_type".

The result? A nicely formatted .csv file that's easily imported in to Google Groups:

Group Email [Required], Member Email, Member Type, Member Role, Member Name
,<imalegend@example.com>,USER,MEMBER,Stephen
,<imaboygenius@example.com>,USER,MEMBER,Stephen
,<imapc@example.com>,USER,MEMBER,Steven
,<imastonecoldsunofagun@example.com>,USER,MEMBER,Steve
,<imamac@example.com>,USER,MEMBER,Steven
,<imawonder@example.com>,USER,MEMBER,Stevland.

Next, one could iterate over all of their lists using the snippets provided, and "Automate the Boring Stuff" without a line of Python. We did exactly that here.

Cheers.

In his book: "Automating the Boring Stuff", Al Sweigart details how to automate boring, repetitive digital tasks using the Python programming language. A job that could take hours of manual work, he explains, can be completed by a computer program faster--and with greater accuracy--than a human being.

Python can add a layer of complexity to a pipeline's automation, though. For example, Python frequently requires the import of one or more modules to achieve a task that a command-line interpreter such as Bash can do directly.

Let's look at a real world example: migrating scores of mailing lists from a decrepit Sendmail server to Google Groups. Our source material takes the following form (abridged):

# List-o-Steves
#
Stephen Gary Wozniak <imalegend@example.com>
Stephen William Hawking <imaboygenius@example.com>
Steven Anthony Ballmer <imapc@example.com>
Steve Austin <imastonecoldsunofagun@example.com>
Steven Paul Jobs <imamac@example.com>
Stevland Hardaway Morris <imawonder@example.com>,

while Google Groups requires formatted .csv files that look like this:

Header: Group Email [required],Member Email,Member Type,Member Role
Entry: yourgroup@email.com, membername@email.com,USER,MEMBER.

For reference:
.

The task, then, is to extract data from our source, and transform that data to a format suitable for import in to Google Groups. But how?

Well, let's start by declaring our variables:

domain="example.com"
file_type="csv"
file="list-o-steves"

and creating a .csv file with appropriately-named headers:

printf "%s\n" "Group Email [Required], Member Email, Member Type, Member Role, Member Name" > "$file"."$file_type"

which returns this file:

list-o-steves.csv

with the following on its first line:

Group Email [Required], Member Email, Member Type, Member Role, Member Name

Progress! We'll cover the nuts-and-bolts of extracting our data in a coming post.

Cheers.

In a previous post we detailed how to archive data "on-the-fly" from a local source to a remote destination, using the utilities tar, ssh, and cat. Today, we'll add nohup to that pipeline so that it can continue uninterrupted, irrespective of your current connection's status (a broken SSH session, e.g.).

Here goes:

First, preface your pipeline with the nohup command:

nohup tar --gzip --create --verbose --file - foobar | ssh foo@foobarserver "cat > /path/to/foobar/foobar.tgz"

Second, stop the command using the following keyboard combination:

CTRL+Z

Third, place this foreground process into the background with:

bg.

You'll then receive output similar to the following in your shell:

PID 26935: completed
PID 8072: in progress (foobar.tgz).

Pay attention to the latter process ID (PID), as that will be the number you'll reference when checking on the job's status using, e.g.: top -p 8072. For an overview of jobs, and background/foreground processes see this article from Red Hat.

Cheers.

Setup Sendmail in GNU/Linux

Scope: Set up a general-purpose internetwork routing facility that supports Simple Mail Transfer Protocol (SMTP).

Dependencies

Install these packages first, per your operating system (OS):

Disable Sendmail daemon

Improve Sendmail security by turning it off.

systemctl stop sendmail

systemctl disable sendmail

Preflight checks

You'll need these.

• Check for required files:

ls /etc/mail/submit.*

should yield:

/etc/mail/submit.cf

/etc/mail/submit.mc

• Check for m4 processor:

which m4

should yield:

/usr/bin/m4

Build submit.cf file

The secret sauce.

• Backup submit.cf:

cp -pv /etc/mail/submit.cf /etc/mail/submit.cf.bak

• Create mysubmit.mc macro configuration file by copying default macro configuration file:

cp -pv submit.mc mysubmit.mc

• Modify mysubmit.mc,

replacing:

[127.0.0.1]

with:

mailhost

on the msp line so that is looks like this:

FEATURE(`msp', `mailhost')dnl`

• Commit your change (from /etc/mail):

m4 mysubmit.mc > /etc/mail/submit.cf

Modify /etc/hosts file

More goodness.

• List the host's IP & hostname, e.g.:

IPaddress hostname alias

• List an SMTP client:

IPaddress hostname alias

Cheers.

For the risk-averse among us, here are a few aliases that can reside in your shell initialization as insurance against the accidental removal of a file(s) or folder(s):

alias cp="cp -i"
alias mv="mv -i"
alias rm="rm -i".

Simply add them to your ~/.bash_profile with the following:

printf '%s\n' 'alias cp="cp -i"' 'alias mv="mv -i"' 'alias rm="rm -i"' >>~/.bash_profile,

being sure to reload your Bash profile by doing: . ~/.bash_profile

Now, when calling the rm command, for example, rather that carelessly expunging your data, you'll receive an interactive prompt asking you if you want to remove a given file. You can then accept or decline the action as shown here:

Aliases need not be limited to "defensive" parameters, though. In fact, they are a useful tool that can save you keystrokes, and by extenstion, time.

Cheers.

One benefit of working from a terminal emulator is the ability to chain commands together in a way that would be difficult to do efficiently from a GUI. For example, the following snippet allows for "on the fly" compression and transfer of a local directory to a remote location:

tar --gzip --create --verbose --file - foobar | ssh foo@foobarserver "cat > /path/to/foobar/foobar.tgz"

Here tar calls the gzip utility to create an archive of the the "foobar" directory, and then pipes (|) the compressed data over a secure shell (ssh) connection. As the data reaches the remote location, it is fed to cat and redirected > to the tarball: "foobar.tgz".

What's nice about this item is that you don't need to tar your data in its entirety before you transfer it; rather, as your data is compressed, it is transferred, then appended, to the remote archive in one fell swoop. This is particularly useful if your local node is running low on disk space and you want to free up space in a hurry. Try doing that with your GUI.

Cheers.

Here's a quick code snippet for use in extracting the current highest
user identifier (UID) in GNU/Linux using awk to do so. For reference: a UID is the unique value your UNIX-like operating system (OS) uses to identify an account and, when used in combination with other components of the OS, determines what system resources said account may access.

UIDs are stored in the world-readable /etc/passwd file, described thoroughly by The Linux Information Project (LINFO) here. In a given entry (e.g.:
linus:x:1001:1001::/home/linus:/bin/bash), each of the seven (7) fields (name, password, user ID, group ID, gecos, home directory, shell) are separated by a colon (:) with no spaces.

To pull a UID from the /etc/passwd file via awk, then, one would use the -F flag and set a colon (:) as the field separator, printing the third column ({print $3}), provided the condition exists. As we're looking for the highest UID, we pipe our results | tail -1 to pull the very last entry from /etc/passwd. With that number in hand, we can simply add to it with a +1.

Take a look:

get_uid() {
  uid=$(awk -F ":" '/1/ {print $3}' /etc/passwd |tail -1)
  increment_uid=$((uid +1))

  printf "%s\\n" "Assigning uid:$increment_uid to $username"
}

Cheers.

In a previous post, we shared a Bash shell script for creating local user accounts in GNU/Linux. As is often the case, the process of drafting this script yielded a number of useful code snippets, one of which we'll detail in this post.

By default, the useradd command will not create the directory structure that GUI users of GNU/Linux distributions may be familiar with (e.g.: Desktop, Music, and the like). While this isn't necessarily a deal-breaker, there may be use cases where this structure is desirable.

To create the default directory structure, we need to call on the xdg-user-dirs tool (for reference: https://wiki.archlinux.org/index.php/XDG_user_directories). Here's one way to do so:

#!/usr/bin/env bash

username='sjobs'

create_default_dirs () { 
  if [[ -n $(command -v xdg-user-dirs-update) ]]
  then
  su ${username} -c xdg-user-dirs-update  
  $username -i xdg-user-dirs-update
fi
} 

In our snippet, we ask Bash to print the path of the xdg-users-dirs-update command (command -v), checking that the length of this check is not zero (0). Once confirmed, we then run the xdg-... command as the user (sjobs, in this case) of the account our script created (su ${username} -c), such that the default directory structure is placed in this user's home folder.

Cheers.

For those running multi-user GNU/Linux instances not bound to a network identity management tool, there are a number of ways to provision local user accounts. As we're a fan of automating repetitive tasks such as the aforementioned, we'll examine how to handle silent password creation when adding users via a shell script.

When creating a user account, we need, at a minimum, a username and password. Here's a function to probe for a password:

get_password () { 

  while true
  
  do      
    read -r -s -p "Enter password to add and press [Enter]: " pass1 
    printf "\\n" 
    read -r -s -p "Re-enter password to add and press [Enter]: " pass2 
    printf "\\n" 

    if [[ "$pass1" != "$pass2" ]]; then 
      printf "%s\n" "ERROR: Passwords do no match."
    else 
      printf "%s\n" "Passwords match. Continuing..."
      break
    fi 
  done
} 

In this snippet, we use a while loop to prompt for a password twice, then check that both user-provided inputs match. Note the read flags: -r (to prevent the \ character from acting as an escape character; -s (to pass input silently, i.e. not print the password to standard output), and -p (to display the prompt).

Having quietly retrieved the password, we next need to create the user account, and then silently set a password for the same. Here's a function for doing so:

set_password_linux() { 

  printf "%s\\n" "Setting password..." 

  printf "%s" "$username:$pass2" | chpasswd 
}

We use the printf command with the stored variables for the username and password, then pipe (|) that information to the chpasswd command. In this way, we've securely obtained and pushed out a password that never appears on screen.

For the full-monty, see here.

Cheers.

In a previous post, we discussed HTTP response codes, and shared a snippet for using curl to extract a website's response code for the purpose of checking if that site was accessible. Today, we'll show how to use wget to achieve the same effect.

Here's the gist of how to do this:

#!/usr/bin/env bash

url="https://geekberg.info"

wget_check() { 

  status_code=$(wget --spider --server-response $url 2>&1 | awk  '/HTTP\/1.1/{print $2}' | head -1)  

  printf "%s\\n" "$status_code"

  if [ "$status_code" -ne "200" ] ; then 
    printf "%s\\n" "BAD URL" 
  else 
    printf "%s\\n" "GOOD URL" 
fi 
} 

wget_check

We use wget's built-in ability to function as a web crawler so that it checks--but does not download--a given site's headers:

URL transformed to HTTPS due to an HSTS policy
Spider mode enabled. Check if remote file exists.
--2019-05-17 00:26:59--  https://geekberg.info/
Resolving geekberg.info (geekberg.info)... 159.65.178.169
Connecting to geekberg.info (geekberg.info)|159.65.178.169|:443... connected.
HTTP request sent, awaiting response... 
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Fri, 17 May 2019 00:26:59 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 13112
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"3338-S/XfSVjCnN/lfdMnlmt2u8WQ478"
Vary: Accept-Encoding
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Length: 13112 (13K) [text/html]

Then, using awk, we parse out the second field of the line beginning with HTTP/1.1 and pipe this information to the head command. This is done because websites can return multiple response codes, and we're only concerned with the first.

From there, we use an if/else constuct to check if the response code from the site is not equal to 200. If that's the case, we'll call it a "BAD URL", otherwise we know it's kosher, hence the "GOOD URL".

Cheers.