notebooks/01.05-IPython-And-Shell-Commands.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython and Shell Commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When working interactively with the standard Python interpreter, one of the frustrations is the need to switch between multiple windows to access Python tools and system command-line tools.\n",
    "IPython bridges this gap, and gives you a syntax for executing shell commands directly from within the IPython terminal.\n",
    "The magic happens with the exclamation point: anything appearing after `!` on a line will be executed not by the Python kernel, but by the system command line.\n",
    "\n",
    "The following discussion assumes you're on a Unix-like system, such as Linux or macOS.\n",
    "Some of the examples that follow will fail on Windows, which uses a different type of shell by default, though if you use the *Windows Subsystem for Linux* the examples here should run correctly.\n",
    "If you're unfamiliar with shell commands, I'd suggest reviewing the [Unix shell tutorial](http://swcarpentry.github.io/shell-novice/) put together by the always excellent Software Carpentry Foundation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Introduction to the Shell\n",
    "\n",
    "A full introduction to using the shell/terminal/command line is well beyond the scope of this chapter, but for the uninitiated I will offer a quick introduction here.\n",
    "The shell is a way to interact textually with your computer.\n",
    "Ever since the mid-1980s, when Microsoft and Apple introduced the first versions of their now ubiquitous graphical operating systems, most computer users have interacted with their operating systems through the familiar menu selections and drag-and-drop movements.\n",
    "But operating systems existed long before these graphical user interfaces, and were primarily controlled through sequences of text input: at the prompt, the user would type a command, and the computer would do what the user told it to.\n",
    "Those early prompt systems were the precursors of the shells and terminals that most data scientists still use today.\n",
    "\n",
    "Someone unfamiliar with the shell might ask why you would bother with this, when many of the same results can be accomplished by simply clicking on icons and menus.\n",
    "A shell user might reply with another question: why hunt for icons and menu items when you can accomplish things much more easily by typing?\n",
    "While it might sound like a typical tech preference impasse, when moving beyond basic tasks it quickly becomes clear that the shell offers much more control of advanced tasks—though admittedly the learning curve can be intimidating.\n",
    "\n",
    "As an example, here is a sample of a Linux/macOS shell session where a user explores, creates, and modifies directories and files on their system (`osx:~ $` is the prompt, and everything after the `$` is the typed command; text that is preceded by a `#` is meant just as description, rather than something you would actually type in):\n",
    "\n",
    "```bash\n",
    "osx:~ $ echo \"hello world\"             # echo is like Python's print function\n",
    "hello world\n",
    "\n",
    "osx:~ $ pwd                            # pwd = print working directory\n",
    "/home/jake                             # This is the \"path\" that we're sitting in\n",
    "\n",
    "osx:~ $ ls                             # ls = list working directory contents\n",
    "notebooks  projects \n",
    "\n",
    "osx:~ $ cd projects/                   # cd = change directory\n",
    "\n",
    "osx:projects $ pwd\n",
    "/home/jake/projects\n",
    "\n",
    "osx:projects $ ls\n",
    "datasci_book   mpld3   myproject.txt\n",
    "\n",
    "osx:projects $ mkdir myproject          # mkdir = make new directory\n",
    "\n",
    "osx:projects $ cd myproject/\n",
    "\n",
    "osx:myproject $ mv ../myproject.txt ./  # mv = move file. Here we're moving the\n",
    "                                        # file myproject.txt from one directory\n",
    "                                        # up (../) to the current directory (./).\n",
    "osx:myproject $ ls\n",
    "myproject.txt\n",
    "```\n",
    "\n",
    "Notice that all of this is just a compact way to do familiar operations (navigating a directory structure, creating a directory, moving a file, etc.) by typing commands rather than clicking icons and menus.\n",
    "With just a few commands (`pwd`, `ls`, `cd`, `mkdir`, and `cp`) you can do many of the most common file operations, but it's when you go beyond these basics that the shell approach becomes really powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shell Commands in IPython\n",
    "\n",
    "Any standard shell command can be used directly in IPython by prefixing it with the `!` character.\n",
    "For example, the `ls`, `pwd`, and `echo` commands can be run as follows:\n",
    "\n",
    "```ipython\n",
    "In [1]: !ls\n",
    "myproject.txt\n",
    "\n",
    "In [2]: !pwd\n",
    "/home/jake/projects/myproject\n",
    "\n",
    "In [3]: !echo \"printing from the shell\"\n",
    "printing from the shell\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passing Values to and from the Shell\n",
    "\n",
    "Shell commands not only can be called from IPython, but can also be made to interact with the IPython namespace.\n",
    "For example, you can save the output of any shell command to a Python list using the assignment operator, `=`:\n",
    "\n",
    "```ipython\n",
    "In [4]: contents = !ls\n",
    "\n",
    "In [5]: print(contents)\n",
    "['myproject.txt']\n",
    "\n",
    "In [6]: directory = !pwd\n",
    "\n",
    "In [7]: print(directory)\n",
    "['/Users/jakevdp/notebooks/tmp/myproject']\n",
    "```\n",
    "\n",
    "These results are not returned as lists, but as a special shell return type defined in IPython:\n",
    "\n",
    "```ipython\n",
    "In [8]: type(directory)\n",
    "IPython.utils.text.SList\n",
    "```\n",
    "\n",
    "This looks and acts a lot like a Python list but has additional functionality, such as\n",
    "the `grep` and `fields` methods and the `s`, `n`, and `p` properties that allow you to search, filter, and display the results in convenient ways.\n",
    "For more information on these, you can use IPython's built-in help features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Communication in the other direction—passing Python variables into the shell—is possible using the `{varname}` syntax:\n",
    "\n",
    "```ipython\n",
    "In [9]: message = \"hello from Python\"\n",
    "\n",
    "In [10]: !echo {message}\n",
    "hello from Python\n",
    "```\n",
    "\n",
    "The curly braces contain the variable name, which is replaced by the variable's contents in the shell command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shell-Related Magic Commands\n",
    "\n",
    "If you play with IPython's shell commands for a while, you might notice that you cannot use `!cd` to navigate the filesystem:\n",
    "\n",
    "```ipython\n",
    "In [11]: !pwd\n",
    "/home/jake/projects/myproject\n",
    "\n",
    "In [12]: !cd ..\n",
    "\n",
    "In [13]: !pwd\n",
    "/home/jake/projects/myproject\n",
    "```\n",
    "\n",
    "The reason is that shell commands in the notebook are executed in a temporary subshell that does not maintain state from command to command.\n",
    "If you'd like to change the working directory in a more enduring way, you can use the `%cd` magic command:\n",
    "\n",
    "```ipython\n",
    "In [14]: %cd ..\n",
    "/home/jake/projects\n",
    "```\n",
    "\n",
    "In fact, by default you can even use this without the `%` sign:\n",
    "\n",
    "```ipython\n",
    "In [15]: cd myproject\n",
    "/home/jake/projects/myproject\n",
    "```\n",
    "\n",
    "This is known as an *automagic* function, and the ability to execute such commands without an explicit `%` can be toggled with the `%automagic` magic function.\n",
    "\n",
    "Besides `%cd`, other available shell-like magic functions are `%cat`, `%cp`, `%env`, `%ls`, `%man`, `%mkdir`, `%more`, `%mv`, `%pwd`, `%rm`, and `%rmdir`, any of which can be used without the `%` sign if `automagic` is on.\n",
    "This makes it so that you can almost treat the IPython prompt as if it's a normal shell:\n",
    "\n",
    "```ipython\n",
    "In [16]: mkdir tmp\n",
    "\n",
    "In [17]: ls\n",
    "myproject.txt  tmp/\n",
    "\n",
    "In [18]: cp myproject.txt tmp/\n",
    "\n",
    "In [19]: ls tmp\n",
    "myproject.txt\n",
    "\n",
    "In [20]: rm -r tmp\n",
    "```\n",
    "\n",
    "This access to the shell from within the same terminal window as your Python session lets you more naturally combine Python and the shell in your workflows with fewer context switches."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}