use grep -E, add Makefile, add -S, -A, -T options, fix ensure_deps.sh for macOS, fix command line parsing, make JSONPath.sh and test scripts robust (#16)

* change use of egrep to grep -E

While we disagree with the notion that `egrep` is obsolete,
tools such as gnu egrep whine when used with the warning:

```
egrep: warning: egrep is obsolescent; using ggrep -E
```

Sadly this pull request changes use of `egrep` to `grep -E`
in order to stop the warnings that gnu egrep now issues.

* add makefile to install

Add a Makefile so that `make install` will work.

Updated `README.md` to mention the `make install` alternative.
Removed trailing whitespace from `README.md`.

* add -S, -A, -T options

We add a `-S` option to print spaces around :s'.

The default (w/o -S):

```json
    "submit_slot":2,
```

With `-S`:

```json
    "submit_slot" : 2,
```

We add an `-A` option start a JSON array on line as JSON member.

The default (w/o -A):

```json
    "authors":
    [
```

With `-A`:

```json
    "authors":[
```

With `-S -A`:

```json
    "authors" : [
```

We add `-T` to indent with tabs.

The default (w/o -T):

```json
    "submit_slot":2,
    "author_count":3,
```

With `-T`:

```json
        "submit_slot":2,
        "author_count":3,
```

We also removed a few trailing whitespace from the shell script.

* fix homebrew paths for macOS in ensure_deps.sh

For macOS (Darwin) we use use the correct paths for homebrew
in `ensure_deps.sh`  If there is an error under macOS, we
also remind the user what to put in their $PATH.

* fix and improve command line parsing and usage

The command line parsing now uses the bash builtin `getopts`.  Now, and
invalid option is flagged as an error.  Improper arguments to command
line options are flagged as errors.  The optional pattern is also obtained
after parsing command line options.

The usage message has been updated to include command line options that
were implemented but not documented in the usage message.

Updated the Invocation section of `README.md` to reflect the current
command line, including command line options that were documented in
the usage message, but not in `README.md`.

* fix test command lines

The original usage message and `README.md` invocation has the optional
pattern at the end of the command line .  We fix 3 tests to place the
pattern at the end of the command line.  Moreover, we put `--` between
command line options and a pattern to prevent the pattern from being
interpreted as a command line option.  And while unlikely to happen,
this would allow pattern that begins with a dash to be correctly parsed.

Ran `all-tests.sh` to test the above.

* improve test scripts

We ran into some issues with the test scripts that were fixed by some
shell script hardening and using some bash best practices.  What follows
are fixes and improvements to the test scripts.

We make the tests scripts more robust:

* detect when the cd fails
* prevent unquoted shell variable expansion from corrupting execution

We improve the use of bash:

* use `env(1)` in the initial `#!` line to find `bash(1)` on `$PATH`

NOTE: macOS has deprecated the distributed `bash(1)` in favor of `zsh(1)`.
      Not all systems have `bash(1)` as `/bin/sh`.
      We allow for `bash(1)` to come from home brew or mac ports, or ...

We use bash constructs for slight speed gain and best practices:

* increment using ((++i)), in some cases avoiding a sub-shell launch
* bypass use of `cat(1)` by using `$(< "$argpfile")`
* use `find(1)` to select test files instead of the slower `ls(1)`
* avoid using shell glob problems by using `find(1)` to select test files
* use `$(...)` instead of `\`cmd\`` to help avoid data corrupting execution

Use `python3(1)` instead of just `python(1)`.

NOTE: Not all systems that have `python3(1)` have `python(1)`.
      Some system admins have yet to select the default version of python.
      The macOS platform, while they have `python3(1)` to not have `python(1)`.

The test scripts now pass the shellcheck script analysis tool.
See [www.shellcheck.net](https://www.shellcheck.net) for details.

Fixed a bug in `all-docker-tests.sh` where Centos tests were re-displayed
a 2nd time as if they were Debian tests.

Ran `./all-tests.sh` to verify the above under macOS and RHEL 9.

TODO: Fix a number of shell script problems with `JSONPath.sh` along
      similar lines as above: especially where shell variables containing
      JSON and other test data can take on values that can
      corrupt the shell script execution.  We wanted to make the
      other test scripts more robust before taking on `JSONPath.sh`
      so that we may use such tests.

* improve JSONPath.sh

We make `JSONPath.sh` much more robust and use bash constructs for slight
speed gain and best practices:

* prevent unquoted shell variable expansion from corrupting execution
* compute numerical values using ((var=equation)) instead of let
* declare local variable before assigning a value
* fixed how options to grep are accumulated
* fixed how main is called
* fixed how an unexpected EOF with token is thrown
* fixed how array elements are unset
* fixed how code determines of an array contains only empty values
* changed cases where `cat(1)` of a file is piped into feeding the file on stdin
* use `$(...)` instead of `\`cmd\`` to help avoid data corrupting execution
* replaced use of `[ ... ]` with `[[ ... ]]`
* removed shell variables that were set but never unused
* fixed cases where a case statement lacked a default case
* fixed how the literal string `'"*'` is used
* fixed cases where a scalar is followed by a literal string that starts with `[`
* fixed cases where `$@` is confused with `$*`

All errors, warnings and issues reported by [shellcheck](https://www.shellcheck.net)
have been addressed / fix.  Now `JSONPath.sh` is error, warning and issue free
under [shellcheck](https://www.shellcheck.net)!

Ran `./all-tests.sh` to verify the above under macOS and RHEL 9 Linux.

Author: lcn2

JSONPath.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env make
+#
+# JSONPath.sh -  JSONPath implementation written in Bash
+
+#############
+# utilities #
+#############
+
+INSTALL= install
+SHELL= bash
+
+######################
+# target information #
+######################
+
+DESTDIR= /usr/local/bin
+
+TARGETS= JSONPath.sh
+
+######################################
+# all - default rule - must be first #
+######################################
+
+all: ${TARGETS}
+
+#################################################
+# .PHONY list of rules that do not create files #
+#################################################
+
+.PHONY: all configure clean clobber install
+
+###################################
+# standard Makefile utility rules #
+###################################
+
+configure:
+
+clean:
+
+clobber: clean
+
+install: all
+	${INSTALL} -m 0555 ${TARGETS} ${DESTDIR}

Makefile

@@ -12,32 +12,50 @@ try wrapping the command like `./ensure_deps.sh ./JSONPath.sh`.
 
 ## Invocation
 
-    JSONPath.sh [-b] [-i] [-j] [-h] [-p] [-u] [-f FILE] [pattern]
+    JSONPath.sh [-h] [-b] [-j] [-u] [-i] [-p] [-w] [-f FILE] [-n] [-s] [-S] [-A] [-T] [pattern]
 
-pattern
-> the JSONPath query. Defaults to '$.\*' if not supplied.
+-h
+> Show help text.
 
 -b
 > Brief output. Only show the values, not the path and key.
 
--f FILE
-> Read a FILE instead of reading from standard input.
-
--i
-> Case insensitive searching.
-
 -j
 > Output in JSON format, instead of JSON.sh format.
 
 -u
 > Strip unnecessary leading path elements.
 
+-i
+> Case insensitive searching.
+
 -p
 > Pass JSON.sh formatted data through to the JSON parser only. Useful after
 > JSON.sh data has been manipulated.
 
--h
-> Show help text.
+-w
+> Match whole words only (for filter script expression).
+
+-f FILE
+> Read a FILE instead of reading from standard input.
+
+-n
+> Do not print header.
+
+-s
+> Normalize solidus.
+
+-S
+> Print spaces around :'s.
+
+-A
+> Start array on same line as JSON member.
+
+-T
+> Indent with tabs instead of 4 character spaces.
+
+pattern
+> the JSONPath query. Defaults to '$.\*' if not supplied.
 
 ## Requirements
 
@@ -55,6 +73,10 @@ Install with npm:
 
 * `sudo npm install -g jsonpath.sh`
 
+Install with make:
+
+* `make install`
+
 Or copy the `JSONPath.sh` script to your PATH, for example:
 
 ``` bash
@@ -215,7 +237,7 @@ done
     -f test/valid/goessner.net.expanded.json \
     '$.store.book[?(@.price<4.20)].[title,price]'
 
-# The following does not work yet (TODO) 
+# The following does not work yet (TODO)
 ./JSONPath.sh \
     -f test/valid/goessner.net.expanded.json \
     '$.store.book[(@.length-1)].title'
@@ -340,7 +362,7 @@ Show all authors, without showing duplicates and output in JSON format.
 All authors with duplicates:
 
 ```
-$ ./JSONPath.sh -f test/valid/goessner.net.expanded.json '$..author' 
+$ ./JSONPath.sh -f test/valid/goessner.net.expanded.json '$..author'
 ... omitted ...
 ["store","book",9,"author"]     "James S. A. Corey"
 ["store","book",10,"author"]    "James S. A. Corey"
@@ -352,7 +374,7 @@ Use standard unix tools to remove duplicates:
 
 ```
 $ ./JSONPath.sh -f test/valid/goessner.net.expanded.json '$..author' \
-    | sort -k2 | uniq -f 1 
+    | sort -k2 | uniq -f 1
 ... 11 lines of output ...
 ```
 
@@ -388,7 +410,7 @@ $ ./JSONPath.sh -f test/valid/goessner.net.expanded.json \
     "book":
     [
         {
-            "author":"Douglas E. Richards" 
+            "author":"Douglas E. Richards"
         },
         {
             "author":"Evelyn Waugh"

README.md

@@ -1,45 +1,46 @@
+#!/usr/bin/env bash
+
 log=testlog.log
 
 # Fedora
 export IMAGE=json-path-fedora-bash
 cp test/docker/Dockerfile-fedora test/docker/Dockerfile
-./test/docker/wrap_in_docker.sh ./all-tests.sh | tee $log
+./test/docker/wrap_in_docker.sh ./all-tests.sh | tee "$log"
 
-a=$(grep 'test(s) failed' $log)
+a=$(grep 'test(s) failed' "$log")
 
 # Ubuntu
 export IMAGE=json-path-ubuntu-bash
 cp test/docker/Dockerfile-ubuntu test/docker/Dockerfile
-./test/docker/wrap_in_docker.sh ./all-tests.sh | tee $log
+./test/docker/wrap_in_docker.sh ./all-tests.sh | tee "$log"
 
-b=$(grep 'test(s) failed' $log)
+b=$(grep 'test(s) failed' "$log")
 
 # Centos
 export IMAGE=json-path-centos-bash
 cp test/docker/Dockerfile-centos test/docker/Dockerfile
-./test/docker/wrap_in_docker.sh ./all-tests.sh | tee $log
+./test/docker/wrap_in_docker.sh ./all-tests.sh | tee "$log"
 
-c=$(grep 'test(s) failed' $log)
+c=$(grep 'test(s) failed' "$log")
 
 # Debian
 export IMAGE=json-path-debian-bash
 cp test/docker/Dockerfile-debian test/docker/Dockerfile
-./test/docker/wrap_in_docker.sh ./all-tests.sh | tee $log
+./test/docker/wrap_in_docker.sh ./all-tests.sh | tee "$log"
 
-d=$(grep 'test(s) failed' $log)
+d=$(grep 'test(s) failed' "$log")
 
 # Cleanup
-rm $log
+rm -- "$log"
 rm test/docker/Dockerfile
 
 # Results
 echo
 echo "Fedora tests"
-echo $a
+echo "$a"
 echo "Ubuntu tests"
-echo $b
+echo "$b"
 echo "Centos tests"
-echo $c
+echo "$c"
 echo "Debian tests"
-echo $c
-
+echo "$d"

all-docker-tests.sh

@@ -1,33 +1,39 @@
-#!/bin/sh
+#!/usr/bin/env bash
 
-cd ${0%/*}
+shopt -s lastpipe
+export CD_FAILED=
+cd "${0%/*}" || CD_FAILED="true"
+if [[ -n $CD_FAILED ]]; then
+   echo "$0: ERROR: cannot cd ${0%/*}" 1>&2
+   exit 1
+fi
 
 #set -e
 fail=0
 tests=0
 #all_tests=${__dirname:}
 #echo PLAN ${#all_tests}
-for test in test/*.sh ;
+find test -mindepth 1 -maxdepth 1 -name '*.sh' -print | while read -r test;
 do
-  tests=$((tests+1))
-  echo TEST: $test
-  ./$test
+  ((++tests))
+  echo TEST: "$test"
+  ./"$test"
   ret=$?
-  if [ $ret -eq 0 ] ; then
-    echo OK: ---- $test
-    passed=$((passed+1))
+  if [[ $ret -eq 0 ]]; then
+    echo OK: ---- "$test"
+    ((++passed))
   else
-    echo FAIL: $test $fail
-    fail=$((fail+ret))
+    echo FAIL: "$test" "$fail"
+    ((fail=fail+ret))
   fi
 done
 
-if [ $fail -eq 0 ]; then
+if [[ $fail -eq 0 ]]; then
   echo -n 'SUCCESS '
   exitcode=0
 else
   echo -n 'FAILURE '
   exitcode=1
 fi
-echo   $passed / $tests
-exit $exitcode
+echo "$passed" "/" "$tests"
+exit "$exitcode"

all-tests.sh

@@ -22,18 +22,22 @@ test_gnu_compat() {
         >&2 echo "Make sure you have GNU sed or compatible installed"
     fi
 
-    if [[ $error -gt 0 && is_osx ]]; then
+    if [[ $error -gt 0 ]] && is_osx; then
         >&2 echo "With homebrew you may install necessary deps via \`brew install grep gnu-sed coreutils\`"
+        >&2 echo 'Put /opt/homebrew/bin/gsed early in your PATH'
+        >&2 echo 'Then put /opt/homebrew/bin/ggrep early in your PATH'
+        >&2 echo 'And put /opt/homebrew/var/homebrew/linked/coreutils/libexec/gnubin early in your PATH'
+        >&2 echo "For example: PATH=$PATH"
     fi
 
-    return $error
+    return "$error"
 }
 
 main() {
     if is_osx; then
-        export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
-        export PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"
-        export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
+        export PATH="/opt/homebrew/bin/gsed:$PATH"
+        export PATH="/opt/homebrew/bin/ggrep:$PATH"
+        export PATH="/opt/homebrew/var/homebrew/linked/coreutils/libexec/gnubin:$PATH"
     fi
 
     test_gnu_compat

ensure_deps.sh

@@ -1,24 +1,30 @@
-#!/bin/sh
+#!/usr/bin/env bash
 
-cd ${0%/*}
+shopt -s lastpipe
+CD_FAILED=
+cd "${0%/*}" || CD_FAILED="true"
+if [[ -n $CD_FAILED ]]; then
+   echo "$0: ERROR: cannot cd ${0%/*}" 1>&2
+   exit 1
+fi
 fails=0
 i=0
-tests=`ls flatten/*.argp* | wc -l`
+tests=$(find flatten -name '*.argp*' -print | wc -l)
 echo "1..${tests##* }"
 # Standard flatten tests
-for argpfile in flatten/*.argp*
+find flatten -name '*.argp*' -print | while read -r argpfile;
 do
   input="${argpfile%.*}.json"
   expected="${argpfile%.*}_${argpfile##*.}.flattened"
-  argp=$(cat $argpfile)
-  i=$((i+1))
-  if ! ../JSONPath.sh "$argp" -u < "$input" | diff -u - "$expected" 
+  argp=$(< "$argpfile")
+  ((++i))
+  if ! ../JSONPath.sh -u -- "$argp" < "$input" | diff -u -- - "$expected"
   then
     echo "not ok $i - $argpfile"
-    fails=$((fails+1))
+    ((++fails))
   else
     echo "ok $i - $argpfile"
   fi
 done
 echo "$fails test(s) failed"
-exit $fails
+exit "$fails"

test/flatten-test.sh

@@ -1,23 +1,29 @@
-#!/bin/sh
+#!/usr/bin/env bash
 
-cd ${0%/*}
+shopt -s lastpipe
+CD_FAILED=
+cd "${0%/*}" || CD_FAILED="true"
+if [[ -n $CD_FAILED ]]; then
+   echo "$0: ERROR: cannot cd ${0%/*}" 1>&2
+   exit 1
+fi
 fails=0
 i=0
-tests=`ls valid/*.argp* | wc -l`
+tests=$(find valid -name '*.argp*' -print | wc -l)
 echo "1..${tests##* }"
 # Json output tests
-for argpfile in valid/*.argp*
+find valid -name '*.argp*' -print | while read -r argpfile;
 do
   input="${argpfile%.*}.json"
-  argp=$(cat $argpfile)
-  i=$((i+1))
-  if ! ../JSONPath.sh "$argp" -j < "$input" | python -mjson.tool >/dev/null
+  argp=$(< "$argpfile")
+  ((++i))
+  if ! ../JSONPath.sh -j -- "$argp" < "$input" | python3 -mjson.tool >/dev/null
   then
     echo "not ok $i - $argpfile"
-    fails=$((fails+1))
+    ((++fails))
   else
     echo "ok $i - JSON validated for $argpfile"
   fi
 done
 echo "$fails test(s) failed"
-exit $fails
+exit "$fails"