Working on the Internet

While a browser is the typical means for accessing web sites on the internet, there are many times where connections are needed from the command line.

The most common tool for transferring across HTTP and HTTPS is curl. There are a variety of uses for it, including accessing REST APIs and downloading files.

With no additional flags, curl will execute a GET to the provided URL.

$ curl http://localhost:8080/greeting
Hello World

The -X flag is used to specify a different HTTP request method to use. When submitting data to a URL in this fashion, the body of the request (specified by the -d flag) and one or more headers indicating the content type (headers are passed via the -H flag) are commonly provided. The body may be either included directly inline or read from a file by prefixing the @ character before the value of the -d flag.

$ curl -X POST -d @request.json -H "Content-Type: application/json" http://localhost:8080/upload

By default, curl will output the received result to the terminal, regardless of if it was text or binary. There are times where the result needs to be written to a file, which is specified through the -o flag.

$ curl -o result.txt http://localhost:8080/greeting
$ cat result.txt
Hello World

Alternatively, many terminals also include the wget tool specifically for downloading files.

$ wget http://localhost:8080/hello-world.png
$ ls
hello-world.png

Unpacking Collections of Files

Many sites will package a number of files together for a single download. There are a variety of formats for this packaging, but the two most common are ZIP files and tarballs.

Zip Files

The unzip command unpacks ZIP files, which conventionally end with the extension .zip. Before actually unzipping the file, it is often a good idea to view the contents of the file first using the -l flag.

$ unzip -l project.zip
Archive: project.zip
Length Date Time Name
--------- ---------- ----- ----
0 05-20-2021 16:15 project/
0 05-20-2021 16:15 project/other-file
26 05-20-2021 16:15 project/my-file
--------- -------
26 3 files

The output shows not only the file names, but the directories into which they will be expanded. In this example, all of the files will be put into a directory called project.

$ unzip ../project.zip
Archive: ../project.zip
creating: project/
extracting: project/other-file
extracting: project/my-file

$ ls
project

$ ls project
my-file other-file

Tarballs

Another common packaging format is a tarball. Similar to ZIP files, tarballs contain a number of files. Since they are encoded using a different format, the tar tool is used to unpack them. When using tar, the -f flag is typically used to indicate that the provided argument is the name of the file to unpack.

Again, it is usually a good idea to see what will be unpacked before actually unpacking a tarball. This is done by using the -t argument.

$ tar -tf project.tar
project/
project/other-file
project/my-file

The tarball is unpacked using the -x flag.

$ tar -xf project.tar
$ ls
project
$ ls project
my-file other-file