Java Read & Write Files in Hadoop Example

How to setup maven, read file, write file, upload file, check file exists, list files, delete file & download file in Hadoop using Java.

Let's following the code below.

Maven Dependencies

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>${hadoop.version}</version>
</dependency>

Write / Upload File

Configuration conf = new Configuration();

conf.set("fs.default.name", "hdfs://localhost:9820"); 	// Same as -> etc/hadoop/core-site.xml
FileSystem fs = FileSystem.get(conf);

FileInputStream fis = new FileInputStream("D:/TestFile.txt");		// Local Path
FSDataOutputStream fsdos = fs.create(new Path("/home/TestFile.txt"));	// Hadoop Path

byte buffer[] = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fis.read(buffer)) > 0) {
	fsdos.write(buffer, 0, bytesRead);
}

fis.close();
fsdos.close();

Create Directory / Folder

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820");  // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

fs.mkdirs(new Path("/home/myfolder"), new FsPermission("1777"));   // Chmod code

Write / Upload File

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

FileInputStream fis = new FileInputStream("D:/TestFile.txt");   	// Local Path
FSDataOutputStream fsdos = fs.create(new Path("/home/TestFile.txt"));   // Hadoop Path

byte buffer[] = new byte[1024];
int bytesRead = 0;
while((bytesRead = fis.read(buffer)) > 0) {
	fsdos.write(buffer, 0, bytesRead);
}

fis.close();
fsdos.close();

Read File

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

FSDataInputStream fsdis = fs.open(new Path("hdfs://localhost:9820/home/TestFile.txt"));

OutputStream os = System.out;
byte buffer[] = new byte[1024];
int bytesRead = 0;
while((bytesRead = fsdis.read(buffer)) > 0) {
	os.write(buffer, 0, bytesRead);
}

fsdis.close();
os.close();

List of Files / Directories

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

FileStatus[] fileStatus = fs.listStatus(new Path("/"));
for(FileStatus status : fileStatus) {
	System.out.println(">> " + status.getPath().toString());
}

Set File Permission (chmod)

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

fs.setPermission(new Path("/home/TestFile.txt"), new FsPermission("1777")); // Chmod code

Check File Exists

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

boolean exists = fs.exists(new Path("/home/TestFile.txt"));

Delete File

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

FileSystem fs = FileSystem.get(conf);

fs.delete(new Path("/home/TestFile.txt"), true);

Download File from HttpServletResponse

public void downloadFromServlet(String remoteFile, HttpServletResponse servletResponse) throws IOException {
	Configuration conf = new Configuration();
	conf.set("fs.default.name", "hdfs://localhost:9820"); // Same as -> etc/hadoop/core-site.xml

	FileSystem fs = FileSystem.get(conf);

	FSDataInputStream fsdis = fs.open(new Path(remoteFile));

	OutputStream os = servletResponse.getOutputStream();
	byte buffer[] = new byte[1024];
	int bytesRead = 0;
	while((bytesRead = fsdis.read(buffer)) > 0) {
		os.write(buffer, 0, bytesRead);
	}

	fsdis.close();
	os.close();
}

References :

• CreativeData.Atlassian.net - Java - Read & Write files with HDFS
  
• JavaTips.net - Apache Hadoop File Permission (FsPermission)
  

Setting Robots Tags on Amazon S3 Objects

Robots meta directives (sometimes called "meta tags") are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file directives give bots suggestions for how to crawl a website's pages, robots meta directives provide more firm instructions on how to crawl and index a page's content.

There are two types of robots meta directives: those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as "noindex" and "nofollow" in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.

This tutorial explains how to setting noindex, nofollow, noarchive or another robots tags on Amazon S3 object.

Ruby

s3.putObject({
    ACL: "public-read",
    Body: "hello world",
    Bucket: "my-bucket",
    CacheControl: "public, max-age=31536000",
    ContentType: "text/plain",
    Key: "hello.txt",
    XRobotsTag: "noindex, nofollow"
}, function(err, resp){});

Java

private static final AWSCredentials credentials = new BasicAWSCredentials(
    "<Access Key>",
    "<Secret Key>"
);

private static final AmazonS3 s3client = AmazonS3ClientBuilder
        .standard()
        .withCredentials(new AWSStaticCredentialsProvider(credentials))
        .withRegion(Regions.AP_SOUTHEAST_1)
        .build();

public static final URL upload(String bucketName, String fileKeyName, File file) {
    PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, fileKeyName, file)
        .withCannedAcl(CannedAccessControlList.PublicRead);

    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setCacheControl("public");
    metadata.setHeader("Pragma", "public");
    metadata.setHeader("X-Robots-Tag", "noindex, nofollow, noarchive, noimageindex, nosnippet, noodp, nodir");

    putObjectRequest.setMetadata(metadata);

    s3client.putObject(putObjectRequest);

    return s3client.getUrl(bucketName, fileKeyName);
}

Reference: 

• StackOverflow - Setting noindex on Amazon S3 objects
• Google Developers - Robots Meta Tags
• Moz.com - Robots Meta Directives