File Crawler with Async Processing

Implement a file crawler that can traverse directories and handle long-running file processing asynchronously.

Part 1 - Basic File Listing: Given a file path, return all files under that path. A helper function list_dir(path) is provided.

Part 2 - Scaling Discussion (No Code): Discuss how to scale for very large directory trees.

Part 3 - Async Processing: Implement an async job system where each file takes a long time to process.

Requirements:

• Handle nested directories recursively
• Process files concurrently
• Track progress and handle failures
• Consider memory constraints for large trees

Constraints:

• Directory tree can be arbitrarily deep
• Individual file processing may take seconds to minutes
• Should not block on slow file operations