3. Working with Files and Directories¶
Lesson objectives
- View, search within, copy, move, and rename files. Create new directories.
- Use wildcards (
*) to perform operations on multiple files. - Make a file read only.
- Use the
historycommand to view and repeat recently used commands.
questions
- How can I view and search file contents?
- How can I create, copy and delete files and directories?
- How can I control who has permission to modify a file?
- How can I repeat recently used commands?
Working with Files¶
Our data set: FASTQ files¶
Now that we know how to navigate around our directory structure, let's
start working with our sequencing files. We did a sequencing experiment and
have two results files, which are stored in our untrimmed_fastq directory.
Wildcards¶
Navigate to your untrimmed_fastq directory:
We are interested in looking at the FASTQ files in this directory. We can list all files with the .fastq extension using the command:
The * character is a special type of character called a wildcard, which can be used to represent any number of any type of character.
Thus, *.fastq matches every file that ends with .fastq.
This command:
The wildcard * represents 0 or more of any number or character. This means that if we, for example, put the wildcard at the start of a a file name, it will match that file (and any other potential file that has more characters):
If you want to match 1 exact number or character, you can use instead the ? character
Try these examples and compare the output:
We can also use explore some of the software pre-installed in our environment:
Lists every file in /usr/bin that ends with the characters 11
/usr/bin/cpp-11 /usr/bin/gcc-ar-11 /usr/bin/gcov-11 /usr/bin/lto-dump-11 /usr/bin/x86_64-linux-gnu-gcc-11 /usr/bin/x86_64-linux-gnu-gcc-ranlib-11 /usr/bin/x86_64-linux-gnu-gcov-tool-11
/usr/bin/g++-11 /usr/bin/gcc-nm-11 /usr/bin/gcov-dump-11 /usr/bin/x86_64-linux-gnu-cpp-11 /usr/bin/x86_64-linux-gnu-gcc-ar-11 /usr/bin/x86_64-linux-gnu-gcov-11 /usr/bin/x86_64-linux-gnu-lto-dump-11
/usr/bin/gcc-11 /usr/bin/gcc-ranlib-11 /usr/bin/gcov-tool-11 /usr/bin/x86_64-linux-gnu-g++-11 /usr/bin/x86_64-linux-gnu-gcc-nm-11 /usr/bin/x86_64-linux-gnu-gcov-dump-11
Note that the output displays full paths to files, since each result starts with /.
Exercise
Do each of the following tasks from your current directory using a single
ls command for each:
- List all of the files in
/usr/binthat start with the letter 'c'. - List all of the files in
/usr/binthat contain the letter 'a'. - List all of the files in
/usr/binthat end with the letter 'o'.
Bonus: List all of the files in /usr/bin that contain the letter 'a' or the
letter 'c'.
Hint: The bonus question requires a Unix wildcard that we haven't talked about yet. Try searching the internet for information about Unix wildcards to find what you need to solve the bonus problem.
Solution
ls /usr/bin/c*ls /usr/bin/*a*ls /usr/bin/*o
Bonus:ls /usr/bin/*[ac]*
Command History¶
If you want to repeat a command that you've run recently, you can access previous commands using the up arrow on your keyboard to go back to the most recent command. Likewise, the down arrow takes you forward in the command history.
A few more useful shortcuts:
- Ctrl+C will cancel the command you are writing, and give you a fresh prompt.
- Ctrl+R will do a reverse-search through your command history. This is very useful.
- Ctrl+L or the
clearcommand will clear your screen.
to see a numbered list of recent commands. You can reuse one of these commands directly by referring to the number of that command.
then you could repeat command #260 by entering:
Type ! (exclamation point) and then the number of the command from your history.
You will be glad you learned this when you need to re-run very complicated commands.
For more information on advanced usage of history, read section 9.3 of
Bash manual.
Exercise
Find the line number in your history for the command that listed all the .sh
files in /usr/bin. Rerun that command.
solution
First type history. Then use ! followed by the line number to rerun that command.
Examining Files¶
We now know how to switch directories, run programs, and look at the contents of directories, but how do we look at the contents of files?
One way to examine a file is to print out all of the
contents using the program cat.
Enter the following command from within the untrimmed_fastq directory:
This will print out all of the contents of the SRR098026.fastq to the screen.
Exercise
- Print out the contents of the
~/shell_data/untrimmed_fastq/SRR097977.fastqfile. What is the last line of the file? - From your home directory, and without changing directories,
use one short command to print the contents of all of the files in
the
~/shell_data/untrimmed_fastqdirectory.
Solution
- The last line of the file is
C:CCC::CCCCCCCC<8?6A:C28C<608'&&&,'$. cat ~/shell_data/untrimmed_fastq/*
cat is a terrific program, but when the file is really big, it can
be annoying to use. The program, less, is useful for this
case. less opens the file as read only, and lets you navigate through it. The navigation commands
are identical to the man program.
Some navigation commands in less:
| key | action |
|---|---|
| Space | to go forward |
| b | to go backward |
| g | to go to the beginning |
| G | to go to the end |
| q | to quit |
less also gives you a way of searching through files. Use the
"/" key to begin a search. Enter the word you would like
to search for and press enter. The screen will jump to the next location where
that word is found.
Search and match commands in less:
| key | action |
|---|---|
| / | to intiate a search (type text to search, then press enter) |
| n | show next match |
| N | show previous |
Shortcut: If you hit "/" then "enter", less will repeat
the previous search. less searches from the current location and
works its way forward. Scroll up a couple lines on your terminal to verify
you are at the beginning of the file. Note, if you are at the end of the file and search
for the sequence "CAA", less will not find it. You either need to go to the
beginning of the file (by typing g) and search again using / or you
can use ? to search backwards in the same way you used / previously.
For instance, let's search forward for the sequence TTTTT in our file.
You can see that we go right to that sequence, what it looks like,
and where it is in the file. If you continue to type / and hit return, you will move
forward to the next instance of this sequence motif. If you instead type ? and hit
return, you will search backwards and move up the file to previous examples of this motif.
Exercises
What are the next three nucleotides (characters) after the first instance of the sequence TTTTT quoted above?
Solution
CAC
What is the sequence ID of the last match that contains the sequence TTCAAA ?
(The sequence ID begins with @SRR097977 and is the line above the sequence)
Solution
SRR097977.216
Remember, the man program actually uses less internally and
therefore uses the same commands, so you can search documentation
using "/" as well!
There's another way that we can look at files, and in this case, just look at part of them. This can be particularly useful if we just want to see the beginning or end of the file, or see how it's formatted.
The commands are head and tail and they let you look at
the beginning and end of a file, respectively.
code
@SRR098026.1 HWUSI-EAS1599_1:2:1:0:968 length=35
NNNNNNNNNNNNNNNNCNNNNNNNNNNNNNNNNNN
+SRR098026.1 HWUSI-EAS1599_1:2:1:0:968 length=35
!!!!!!!!!!!!!!!!#!!!!!!!!!!!!!!!!!!
@SRR098026.2 HWUSI-EAS1599_1:2:1:0:312 length=35
NNNNNNNNNNNNNNNNANNNNNNNNNNNNNNNNNN
+SRR098026.2 HWUSI-EAS1599_1:2:1:0:312 length=35
!!!!!!!!!!!!!!!!#!!!!!!!!!!!!!!!!!!
@SRR098026.3 HWUSI-EAS1599_1:2:1:0:570 length=35
NNNNNNNNNNNNNNNNANNNNNNNNNNNNNNNNNN
+SRR098026.247 HWUSI-EAS1599_1:2:1:2:1311 length=35
#!##!#################!!!!!!!######
@SRR098026.248 HWUSI-EAS1599_1:2:1:2:118 length=35
GNTGNGGTCATCATACGCGCCCNNNNNNNGGCATG
+SRR098026.248 HWUSI-EAS1599_1:2:1:2:118 length=35
B!;?!A=5922:##########!!!!!!!######
@SRR098026.249 HWUSI-EAS1599_1:2:1:2:1057 length=35
CNCTNTATGCGTACGGCAGTGANNNNNNNGGAGAT
+SRR098026.249 HWUSI-EAS1599_1:2:1:2:1057 length=35
A!@B!BBB@ABAB#########!!!!!!!######
The -n option to either of these commands can be used to print the first or last n lines of a file.
Details on the FASTQ format¶
Although it looks complicated (and it is), it's easy to understand the fastq format with a little decoding. Some rules about the format include...
| Line | Description |
|---|---|
| 1 | This is the header or sequence identifier line and it always begins with '@' and then information about the read |
| 2 | The actual DNA sequence |
| 3 | Always begins with a '+' and sometimes the same info in line 1 |
| 4 | Has a string of characters which represent the quality scores; must have same number of characters as line 2 |
We can view the first complete read in one of the files in our dataset by using head to look at
the first four lines.
code
All but one of the nucleotides in this read are unknown (N). This is a pretty bad read!
Line 4 shows the quality for each nucleotide in the read. You can read more about the Fastq format on Wikipedia.
Creating, moving, copying, and removing¶
Now we can move around in the file structure, look at files, and search files. But what if we want to copy files or move them around or get rid of them? Most of the time, you can do these sorts of file manipulations without the command line, but there will be some cases (like when you're working with a remote computer like we are for this lesson) where it will be impossible. You'll also find that you may be working with hundreds of files and want to do similar manipulations to all of those files. In cases like this, it's much faster to do these operations at the command line.
Copying Files¶
When working with computational data, it's important to keep a safe copy of that data that can't be accidentally overwritten or deleted. For this lesson, our raw data is our FASTQ files. We don't want to accidentally change the original files, so we'll make a copy of them and change the file permissions so that we can read from, but not write to, the files.
First, let's make a copy of one of our FASTQ files using the cp command.
Navigate to the ~/shell_data/untrimmed_fastq directory and enter:
We now have two copies of the SRR098026.fastq file, one of them named SRR098026-copy.fastq. We'll move this file to a new directory
called backup where we'll store our backup data files.
Creating Directories¶
The mkdir command is used to make a directory. Enter mkdir followed by a space, then the directory name you want to create:
Moving / Renaming¶
We can now move our backup file to this directory. We can move files around using the command mv:
The mv command is also how you rename files. Let's rename this file to make it clear that this is a backup:
Let's say you don't want this backup file any more, you can delete it with:
And if you want to delete the whole directory, you'll need to first move back out of the directory, then run rm with the recursive flag:
This will delete not only the directory, but all files within the directory. If you have write-protected files in the directory, you will be asked whether you want to override your permission settings.
Heads up!
The remove command permanently deletes your file or directory. There is no undo. Use wisely.
keypoints
- You can view file contents using
less,cat,headortail. - The commands
cp,mv, andmkdirare useful for manipulating existing files and creating new directories. - You can view file permissions using
ls -land change permissions usingchmod. - The
historycommand and the up arrow on your keyboard can be used to repeat recently used commands.