Hadoop Distributed File System (HDFS) - Revision history

Sbogaart: /* How to connect to the service */

2025-05-19T08:41:00Z

How to connect to the service

Sbogaart: /* How to connect to the service */

2025-05-14T13:14:30Z

How to connect to the service

Sbogaart: /* Introduction */

2025-05-14T13:13:32Z

Introduction

Sbogaart: The WebDAV service over WebHDFS offered by PIC provides a user-friendly and efficient way to interact with your data on HDFS. Whether you're uploading large files, managing your datasets, or downloading files, this service simplifies the process by integrating with common file management tools.

2025-05-14T12:47:13Z

The WebDAV service over WebHDFS offered by PIC provides a user-friendly and efficient way to interact with your data on HDFS. Whether you're uploading large files, managing your datasets, or downloading files, this service simplifies the process by integrating with common file management tools.

New page

= Introduction =

PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.

WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.

The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.

== How to connect to the service ==
To connect to the WebDAV service, follow the steps below:

1. Use a WebDAV-compatible client:
Finder (macOS)
File Explorer (Windows)
Linux File System (Linux)
rclone (CLI)
Cyberduck (macOS, Windows)
CrossFTP (macOS, Windows, Linux)

2. Mount the WebDAV endpoint:
The WebDAV server is accessible via the following URL: [https://webdav.pic.es/hdfs]

3. Authenticate with your PIC credentials:
When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.

4. Browse, upload, and download files:
Once connected, you will be able to manage your files in the same way you would with any local file system. You can drag and drop files, create directories, and manage large datasets directly on the HDFS.
Tip: Using a multithreaded client like rclone or Cyberduck will help improve upload and download speeds for large files.

==Usage Guidelines==
- File Uploads:
· The WebDAV service allows you to upload files of any size to HDFS.
· For large datasets, it is recommended to use a multithreaded client, as this will optimize the upload process and make it faster and more efficient.
- File Downloads:
· You can also download files from the HDFS to your local machine.
· Multithreading is supported, allowing you to download large files quickly.
- File Management:
· Files and directories can be created, deleted, or renamed directly within your WebDAV-compatible client.
· You can also move files around within your HDFS storage, making file management easier.

==Best Practices==
1. Optimize large file transfers:
To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.

==Troubleshooting==

1. Cannot Connect to the WebDAV Service:
If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.
2. Slow Uploads or Downloads:
If you experience slow file transfer speeds, check if your client supports multithreaded transfers and consider using a tool like rclone or Cyberduck, which support multiple threads for faster transfers.
3. File Uploads Fail:
If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.

==Security and Data Management==

- PIC Credentials:
Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.

@@ Line 25: / Line 25: @@
 #*curl
 # '''Mount the WebDAV endpoint:'''
-#: The WebDAV server is accessible via the following URL: https://webdav.pic.es/ [https://webdav.pic.es/]
+#: The WebDAV server is accessible via the following URL: https://webdav-hdfs.pic.es/ [https://webdav.pic.es/]
 # '''Authenticate with your PIC credentials:'''
 #: When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.

@@ Line 21: / Line 21: @@
 #*Linux File System (Linux)
 #*rclone (CLI)
-#*Cyberduck (macOS, Windows)
+#*Cyberduck
-#*CrossFTP (macOS, Windows, Linux)
+#*CrossFTP
 # '''Mount the WebDAV endpoint:'''
 #: The WebDAV server is accessible via the following URL: https://webdav.pic.es/ [https://webdav.pic.es/]

@@ Line 1: / Line 1: @@
 = Introduction =
 PIC offers a WebDAV service over WebHDFS for seamless file transfer and management. This service is designed for users who need to upload, download, and manage large files on HDFS through a familiar file management interface. The WebDAV protocol allows users to access the HDFS in a way that mimics the experience of working with local file systems, offering the flexibility and ease of use of tools like Finder, File Explorer, and rclone, among others.
 WebDAV-compatible clients allow for the management of large datasets and offer multithreaded operations for enhanced performance. This makes the WebDAV service ideal for handling large-scale data uploads and downloads, with support for efficient file transfers.
 The service also provides a simple and effective way to interface with HDFS, especially for users who prefer a file-system-like experience for managing their data, rather than relying on more technical methods like command-line tools.
-== How to connect to the service ==
+= How to connect to the service =
 To connect to the WebDAV service, follow the steps below:
-. Use a WebDAV-compatible client:
+# '''Use a WebDAV-compatible client:'''
-Finder (macOS)
+#*Finder (macOS)
-File Explorer (Windows)
+#*File Explorer (Windows)
-Linux File System (Linux)
+#*Linux File System (Linux)
-rclone (CLI)
+#*rclone (CLI)
-Cyberduck (macOS, Windows)
+#*Cyberduck (macOS, Windows)
-CrossFTP (macOS, Windows, Linux)
+#*CrossFTP (macOS, Windows, Linux)
-. Mount the WebDAV endpoint:
+** You can also move files around within your HDFS storage, making file management easier.
-. Authenticate with your PIC credentials:
+=Best Practices=
-When prompted, enter your PIC user credentials to authenticate your session. This ensures that only authorized users have access to the HDFS.
+'''Optimize large file transfers:'''
-. Browse, upload, and download files:
+: To improve performance, use tools like rclone or Cyberduck for multithreaded file transfers. This helps manage the upload and download of large files or large quantities of files more efficiently.
-==Usage Guidelines==
+=Troubleshooting=
-==Best Practices==
+'''Cannot Connect to the WebDAV Service:'''
-. Optimize large file transfers:
+:If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.
-==Troubleshooting==
+'''Slow Uploads or Downloads:'''
-. Cannot Connect to the WebDAV Service:
+'''File Uploads Fail'''
-If you're unable to connect to the WebDAV server, ensure that you're using the correct URL and that your PIC credentials are entered properly. Also, verify that your WebDAV-compatible client is configured correctly.
+:If file uploads fail, try splitting large files into smaller chunks or use an alternative tool that supports better error handling and retry capabilities, such as rclone.
-==Security and Data Management==
+=Security and Data Management=
-- PIC Credentials:
+'''PIC Credentials:'''
-Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.
+: Always ensure that your PIC credentials are secure. Do not share your credentials with others or store them in insecure locations.