Hadoop command reference

Below is a list of helpful command references with descriptions for Hadoop.

ls 

Lists the contents of the directory specified by path, showing the names, permissions, owner, size and modification date for each entry.

lsr 

Behaves like -ls, but recursively displays entries in all subdirectories of path.

du 

Shows disk usage, in bytes, for all the files which match path; filenames are reported with the full HDFS protocol prefix.

dus 

Like -du, but prints a summary of disk usage of all files/directories in the path.

mv 

Moves the file or directory indicated by src to dest, within HDFS.

cp  

Copies the file or directory identified by src to dest, within HDFS.

rm 

Removes the file or empty directory identified by path.

rmr 

Removes the file or directory identified by path. Recursively deletes any child entries (i.e., files or subdirectories of path).

put  

Copies the file or directory from the local file system identified by localSrc to dest within the DFS.

copyFromLocal  

Identical to -put

moveFromLocal  

Copies the file or directory from the local file system identified by localSrc to dest within HDFS, and then deletes the local copy on success.

get [-crc]  

Copies the file or directory in HDFS identified by src to the local file system path identified by localDest.

getmerge  

Retrieves all files that match the path src in HDFS, and copies them to a single, merged file in the local file system identified by localDest.

cat 

Displays the contents of filename on stdout.

copyToLocal  

Identical to -get

moveToLocal  

Works like -get, but deletes the HDFS copy on success.

mkdir 

Creates a directory named path in HDFS.
Creates any parent directories in path that are missing (e.g., mkdir -p in Linux).

setrep [-R] [-w] rep 

Sets the target replication factor for files identified by path to rep. (The actual replication factor will move toward the target over time)

touchz 

Creates a file at path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.

test -[ezd] 

Returns 1 if path exists; has zero length; or is a directory or 0 otherwise.

stat [format] 

Prints information about path. Format is a string which accepts file size in blocks (%b), filename (%n), block size (%o), replication (%r), and modification date (%y, %Y).

tail [-f] 

Shows the last 1KB of file on stdout.

chmod [-R] mode,mode,... ...

Changes the file permissions associated with one or more objects identified by path…. Performs changes recursively with R. mode is a 3-digit octal mode, or {augo}+/-{rwxX}. Assumes if no scope is specified and does not apply an umask.

chown [-R] [owner][:[group]] ...

Sets the owning user and/or group for files or directories identified by path…. Sets owner recursively if -R is specified.

chgrp [-R] group ...

Sets the owning group for files or directories identified by path…. Sets group recursively if -R is specified.

help 

Returns usage information for one of the commands listed above. You must omit the leading ‘-‘ character in cmd.