RMount is a lot more robust than existing libraries and methods to create a remote connection to a cloud server. For example you would want to use Python to connect to a cloud storage such as S3 or a remote server via SSH and treat it like a local file-system.

Note

It has been a pain to write this library given how many things can go wrong when treating a cloud storage as a local file-system. For example, connection drops, or writing of large files can make the system unresponsive and lead to loss of data. The library is supposed to take care of such cases but it is always subject to unforeseen failures.

Feel free to try it out

or pip install rmount

Examples

Google Colab

  • iordanis_@fediverser.communick.devOPBEnglish
    0·
    10 months ago

    Primarily they do not monitor the status or health of the connection to the remote file-system. You can end-up writing GBs of data thinking everything is working fine only to realize upon program termination all your data was lost.

    There are no python alternatives that work, while the best method of doing that is using rclone which is written in go (which is the back-end also used for rmount). However the only way to troubleshoot broken connection or issues is by going through the logs of the library and a lot of manual effort (which is what rmount primarily does for you out-of-the box)

    Part of the reason of all the issues mentioned is that rclone uses Fuse file-system and as such from several layers of abstraction (Fuse-> Go-> RClone-> Python) it becomes more difficult to diagnose and deal with failures. That is what rmount tries to do.

    • cmcclu5@fediverser.communick.devBEnglish
      0·
      10 months ago

      What benefits would this have over something like s3fs? I’ve never had issues with that, but it doesn’t cover all the use cases you outlined. There are robust alternatives for SSH as well that have connection error monitoring to avoid the situation you outlined.

      • skrt123@fediverser.communick.devBEnglish
        1·
        10 months ago

        Yep, I mean using aws cli is good enough. A simple aws s3 cp copy_from copy_to is a command that should get 99% of people moving along.

        • cmcclu5@fediverser.communick.devBEnglish
          1·
          10 months ago

          I normally do that as well, but some people prefer to mount s3 as a file system like mentioned in this post. I’ve had to use it for work when my boss required it. I’d personally use the CLI for everything.