Home > cloud servers, Linux > Mounting Rackspace Cloud Files using cloudfuse into ubuntu 10.10 v2

Mounting Rackspace Cloud Files using cloudfuse into ubuntu 10.10 v2

This article shows how to mount cloud files using cloudfuse software into your ubuntu 10.10 as a directory so you can access your cloud files containers data inside your linux server just like any other folder and files. One heads up, this gives you an easy access to your cloud files data but in no way means you can use it as a place for any database/application directly running from it, will be darn slow. So, why would one want it then? Well, there are plenty of uses of having system level access to your cloud files, for instance if you have some scripts which create your mysql backup or website backup, those scripts can create backups automatically into cloud files without you needing to copy them yourself. So, lets get on with it..

Note: The following commands are tried and tested on Ubuntu 10.10, but they should be easily applicable to other versions of ubuntu or debian. As long as you install the required packages, you should be able to compile cloudfuse code and use it.

Installing cloud fuse:
First download the cloudfuse code from cloudfuse-0.1.tar.gz. I downloaded this code from upsteam maintainer and shared it here so that all the below steps work fine, as the new code changes in repository can cause compile time errors. So, download the code from above link . Extract this file and then compile.

ssidhu@ssidhu:~$ tar -xzvf cloudfuse-0.1.tar.gz

Once you have extracted the .tar.gz file, you should have following files under cloudfuse-0.1 directory.

root@ubuntu-test:~/cloudfuse-0.1# ls -la
total 280
drwxr-xr-x 3 root root   4096 Feb 21 21:47 .
drwx------ 4 root root   4096 Feb 21 21:47 ..
drwxr-xr-x 8 root root   4096 Feb 21 21:47 .git
-rw-r--r-- 1 root root   1059 Feb 21 21:47 LICENSE
-rw-r--r-- 1 root root   1024 Feb 21 21:47 Makefile.in
-rw-r--r-- 1 root root   2332 Feb 21 21:47 README
-rw-r--r-- 1 root root  12014 Feb 21 21:47 cloudfsapi.c
-rw-r--r-- 1 root root   1043 Feb 21 21:47 cloudfsapi.h
-rw-r--r-- 1 root root  11240 Feb 21 21:47 cloudfuse.c
-rw-r--r-- 1 root root   4335 Feb 21 21:47 config.h.in
-rwxr-xr-x 1 root root 198521 Feb 21 21:47 configure
-rw-r--r-- 1 root root   1324 Feb 21 21:47 configure.in
-rwxr-xr-x 1 root root  13184 Feb 21 21:47 install-sh
root@ubuntu-test:~/cloudfuse-0.1#

Now its time to compile it and install it. You’ll need libcurl, libfuse, and libxml2 and their dev packages installed to build it.

Cloudfuse is built and installed like any other autoconf-configured code. Normally,

./configure
make
sudo make install

But, first you need to install the required packages, otherwise the ./configure command will fail and throw you errors.

apt-get update
apt-get install gcc
apt-get install libcurl4-openssl-dev
apt-get install libxml2 libxml2-dev
apt-get install libfuse-dev

now run the following command in the cloudfuse directory

root@ubuntu-test:~/cloudfuse-0.1# ./configure
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for a BSD-compatible install... /usr/bin/install -c
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for XML... yes
checking for CURL... yes
checking for FUSE... yes
.........
............
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
root@ubuntu-test:~/cloudfuse-0.1# make
gcc -g -O2 -I/usr/include/libxml2     -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse   -o cloudfuse cloudfsapi.c cloudfuse.c -lxml2   -lcurl   -pthread -lfuse -lrt -ldl
root@ubuntu-test:~/cloudfuse-0.1# make install
/usr/bin/install -c cloudfuse /usr/local/bin/cloudfuse
root@ubuntu-test:~/cloudfuse-0.1#

If everything went fine, you should have cloudfuse installed properly. Confirm this by running the which command. It should show the location of the cloudfuse binary file.

root@ubuntu-test:~/cloudfuse-0.1# which cloudfuse
/usr/local/bin/cloudfuse
root@ubuntu-test:~/cloudfuse-0.1#

Mounting cloudfiles:
Let’s now use cloudfuse and mount our cloudfiles.

You’ll have to create a configuration file for cloudfuse in your home directory and put your Rackspace cloudfiles username and API key in it, like below:

$HOME/.cloudfuse
    username=[username]
    api_key=[api key]
    authurl=[auth URL]

Auth URLs:
US cloudfiles account: https://auth.api.rackspacecloud.com/v1.0
UK cloudfiles account: https://lon.auth.api.rackspacecloud.com/v1.0

The following entries are optional, you can define these values in the .cloudfuse file.

     use_snet=[True to use snet for connections]
     cache_timeout=[seconds for directory caching, default 600]

After creating the above configuration file, you will run the cloudfuse command like following. The syntax should be as simple as:

cloudfuse [mount point]

So, you should be able to mount cloud like this

root@ubuntu-test:/# mkdir cloudfiles
root@ubuntu-test:/# cloudfuse /cloudfiles

If you run # ls -la command inside the /cloudfiles directory you should see your cloudfiles containers.

If you are not the root of the system, then you username will need to be part of “fuse” group. This can probably be accomplished with:

sudo usermod -a -G fuse [username]

If you are unable to see any containers inside the mountpoint, then probably some of the above steps didn’t work properly. You need to check and make sure that all the above steps get completed properly.

UPDATE: 30/9/2011

Here is some extra info for CentOS on how to mount cloudfuse using another use ie. Apache.

$ yum install fuse
$ usermod -a -G fuse apache
$ mkdir /mnt/cloudfiles
$ chown apache:apache /mnt/cloudfiles
$ sudo -u apache cloudfuse /mnt/cloudfiles -o username=myuser,api_key=mykey,use_snet=true,authurl="https://lon.auth.api.rackspacecloud.com/v1.0"

Play around with it and fix it how you like, but I think it would be useful. Courtesy my frnd Anh.

Let me know if there are any errors in these instructions or you faced some difficulty understanding them, I will update them accordingly. Any comments are highly appreciated.

Note: Please don’t bug Rackspace Support for the help on this article, it’s not supported by them hence this article. :)

Good luck!

Categories: cloud servers, Linux
  1. mandm
    March 9, 2011 at 12:07 am | #1

    yes this worked perfectly fine for me, i used the authurl for US cloudfiles and now i can mount my cloud files on the server
    Thanks a lot sandeep for helping me out

  2. Flo
    March 11, 2011 at 12:31 pm | #2

    Hi Sandeep, thanks for your previous replies. Sorry have not had time to investigate yet, but saw you added Auth URLs. It’s looking promising. Many thanks, F.

  3. April 7, 2011 at 4:40 pm | #3

    Sandeep, thanks so much. I’ve been trying to map files to my Cloud Server for a month! Just so you know, I received a link to this article from Rackspace support — they like this a lot.

    • April 14, 2011 at 10:55 pm | #4

      Hi Trenton,

      I’m glad that it worked for you, thanks for the positive feedback! :)

      -Sandeep

  4. Hamid
    April 11, 2011 at 4:29 pm | #5

    Hi Sandeep,

    I have mounted the cloud file to cloud server(Linux) but I acnnot copy files and folders from the cloud server to cloud file.

    I got follwoing error messages:

    Operation not permitted

    Thanks

    Hamid

  5. mandm
    April 14, 2011 at 8:31 pm | #6

    Hi Sandeep, so i build a new server and tried all the steps mentioned in here and i get this error, so i guess this needs to be added to document??

    $ sudo mount /media/cloudfiles
    mount: can’t find /media/cloudfiles in /etc/fstab or /etc/mtab

    Thanks once again for being proactive and putting up this article

    • April 14, 2011 at 10:55 pm | #7

      Hi Mandm,

      You are running the wrong command to mount the cloudfiles. do following as mentioned in the article. Cloud files cannot be mounted using normal mount command, you need to run the “cloudfuse” command as following:

      mkdir /cloudfiles
      cloudfuse /cloudfiles

      hope it works for you.

      Thanks,
      Sandeep

  6. mandm
    April 15, 2011 at 4:07 am | #8

    Hi Sandeep,
    Thanks for pointing out, that works like a charm. So may be that might have been the reason for the segfault issues i had where the servers would drop the mount point and also that there was a lag between both the servers seeing files uploaded from the other one…..
    i’ll keep you posted on what i see

    • mandm
      April 15, 2011 at 4:30 am | #9

      actually sandeep let me take that back, i might have mistyped the issue earlier, i use this command to load my cloud files, since i wanted to read and write to the files

      sudo cloudfuse /media/cloudfiles -o allow_other,nonempty

      so it still drops the mount point periodically and there is a lag when one server sees the files uploaded from other server by about 5 mins, any ideas?

  7. mandm
    April 15, 2011 at 6:56 pm | #10

    hi Sandeep,
    So even on the new server it gives me this error
    kernel: [87106.412129] cloudfuse[19158]: segfault at 1b1 ip 00007f5d78492827 sp 00007f5d73e10558 error 4 in libxml2.so.2.7.6[7f5d783e8000+146000]

    So what verion of libxml2 so you have?
    mine is

    $ sudo dpkg –list libxml2
    Version Description
    +++-=====================-======================-==================================
    ii libxml2 2.7.6.dfsg-1ubuntu1.1 GNOME XML library

  8. Jody Durkacs
    April 20, 2011 at 5:47 pm | #11

    This works great for me, but I can not get it to allow anyone but root to write to this location. Is there a way to mount it read-write for a non-root user? The mount command claims it is mounted rw, but it won’t let other users cd to the directory or copy files there.

  9. mandm
    April 22, 2011 at 7:51 pm | #12

    hi Jody, you should use this command

    sudo cloudfuse /media/cloudfiles -o allow_other,nonempty

    this will help you read write files other than the root user, even thouugh the file ownership and group ownership shows as root root, it is stil accessible via apache user..etc

  10. Jody Durkacs
    April 27, 2011 at 7:19 pm | #13

    That worked, thank you. Now I just need to get automount with /etc/fstab working. It won’t auto-mount at boot. If anyone can help, I am using the following line in that file:
    cloudfuse /home/user/cloud fuse defaults,allow_other,gid=1002,umask=007,username=cloudusername,api_key=XXXXXXXXX 0 0

    Also, if I have that in fstab and I try mount -a I get this output:
    mount: wrong fs type, bad option, bad superblock on cloudfuse,
    missing codepage or helper program, or other error
    (for several filesystems (e.g. nfs, cifs) you might
    need a /sbin/mount. helper program)
    In some cases useful info is found in syslog – try
    dmesg | tail or so

    I think a clue is that after a reboot, if I manually try to mount it, I initially get this message:
    fuse: device not found, try ‘modprobe fuse’ first

    and if I enter ‘modprobe fuse’ and then try to manually mount, it works.

    Can anyone tell me what I need to do to make this work with fstab?

  11. mandm
    April 27, 2011 at 10:00 pm | #14

    Yeah…same issues that i faced…:)

    so i don’t use the /etc/fstab, for mounting the cloudfiles on server reboot try this

    add these entries to the /etc/crontab

    @reboot root modprobe fuse
    @reboot root cloudfuse /media/cloudfiles -o allow_other,nonempty

    and try restarting the server, you should see the cloudfiles mounted

    • April 27, 2011 at 10:43 pm | #15

      thanks mandm, this is really great and makes me so much happy, we all helping each other, that’s what open source is all about.. :) I’ll go ahead add these instructions to the post as well. once again thanks for helping out Jody.

    • June 13, 2011 at 1:47 pm | #16

      I noticed a oddity with cron under CentOS not setting the $HOME variable for the executing user, so the cloudfiles account never gets mounted as it can’t find the $HOME/.cloudfuse file. To get around this we can explicitly specify the $HOME variable to use at runtime,

      @reboot root HOME=/root cloudfuse /media/cloudfiles -o allow_other,nonempty

      Alternative, it is also possible to specify all mount options in the crontab.

  12. mandm
    April 28, 2011 at 5:09 am | #17

    Not a problem Sandeep, i am just trying to share as i learn..:)

  13. Jody Durkacs
    April 28, 2011 at 2:31 pm | #18

    That worked perfectly! Thanks mandm!

  14. mandm
    May 12, 2011 at 8:35 pm | #19

    Hi Sandeep,
    is there a way to get around the 10000 files limit per container ?

    I have hit the limit and the wierd part is that any file or folder created by my application gets deleted if it has letters greater than ‘f’ since i have 70K files which starts with ‘f’

  15. Hal Burgiss
    May 20, 2011 at 5:13 pm | #20

    I have twice tried to use this to sync files as a backup mechanism, and twice it (or something related) has corrupted the local filesystem.

    # ll /mnt
    ls: cannot access /mnt/cloud: Transport endpoint is not connected
    total 8.0K
    drwxr-xr-x 4 root root 4.0K 2011-05-18 10:23 ./
    drwxr-xr-x 24 root root 4.0K 2011-05-16 13:54 ../
    d????????? ? ? ? ? ? cloud/
    drwxr-xr-x 2 root root 0 1969-12-31 19:00 rackspace/

    The rackspace mount point does not look too screwy, but any attempt to access it results in a hung terminal. The cloud one is obviously fubar.

    • mandm
      June 23, 2011 at 2:45 pm | #21

      Yes i agree it is a complete fubar in terms of reliability and scalability. You can only use this in a non productive and an environment where the users have no interaction.
      I have a cronjob running every minute to get around this error and once the jobs sees that the transport endpoint is not connected it umounts and remounts the drive

      But yes, there is no point in using it for scalable and highly visible environments

      (Using it since Jan 2011)

      • Hal Burgiss
        June 23, 2011 at 4:06 pm | #22

        I am having better results just mounting on demand. I am using this for backups and this is done nightly. So I unmount as soon as the backup is completed. That is helping. I have also identified some other “issues” that have caused reliability problems. Copying 2G+ files (at least from a 32bit system) will cause a hang. The same files move easily between local Linux 32bit systems, and to Cloud Servers. And in fact with other tools to Cloud files. I have also found 2 files with very strange character encoding in the filenames, that would cause a dropped connection. There may be some rsync options too that don’t play nice.

  16. Nandeesh
    June 21, 2011 at 7:07 am | #23

    Hi Sandeep,

    Good Work,

    I am new to Linux. trying hard to work with this.. can you guide me few steps to learn Linux as early as possible.

  17. Brad
    July 10, 2011 at 5:43 pm | #24

    Hi – when I try to mount, I get the error:

    root@host [/]# cloudfuse /cloudfiles
    Unable to authenticate.

    I’ve verified the API and username with Rackspace and all are correct. Any idea what could be wrong?

    • July 10, 2011 at 5:48 pm | #25

      Brad, make sure you are using correct API AUTH URL, US and UK cloud files accounts has different URLs. Also, are you using the code which I have shared in the blog or you checked out directly from github? if you checked out directly from github something might have changed in the code. The source code which I have shared with the link in the article work fine with all the instructions, so it might be AUTH URL or make sure your linux has ca-bunble certs installed, test the authentication with cURL first as well.

      good luck!
      Sandeep

  18. Shane T
    July 13, 2011 at 6:51 pm | #26

    Hi Hal… What are you using to get around that “Transport endpoint…” error to unmount the drive? I have the same error and wanted to take your recommendations to mount/unmount on demand but I don’t see any unmount options within cloudfuse?

    -st

  19. Will Sinclair
    July 19, 2011 at 4:30 pm | #27

    Sandeep,

    I am also having the

    “Unable to authenticate.
    Bad username or password” error

    I have followed your instructions meticulously, and triple checked everything is as you say it should be.

    How do I check if my server has ca-bunble?

    • Will Sinclair
      July 19, 2011 at 5:12 pm | #28

      I worked out the issue, in cloudfuse.c you are hardcoding the apiurl option and ignoring the one pulled from ~/.cloudfuse … I am auth’ing against the london server.

      Cheers,
      W

  20. Will Sinclair
    July 19, 2011 at 5:29 pm | #29

    Well i can mount fine, but when trying to access the mountpoint it just hangs, any ideas?

    • Jody Durkacs
      July 19, 2011 at 6:55 pm | #30

      Something similar happens to me. Occasionally my cloudfuse mount will just hang if I try to access it from a shell. If I umount it, I get an error stating it’s busy. This has happened twice in the last couple months. A reboot of the system solved it.

    • July 19, 2011 at 7:02 pm | #31

      Hi Will,

      check with curl, make sure you can authenticate to the AUTH_URL with https and your curl shouldn’t throw any error. If the curl works fine that means ca-bundle is working fine.

      http://www.productionscale.com/home/2009/8/2/using-curl-to-access-the-rackspace-cloud-api.html#axzz1SZyo778r

      and regarding AUTH_URL, did you check out the new code from github or used the one which I’m sharing in this post, the code which I’m sharing works fine with all the instructions on this page and it does check the .cloudfuse configuration and use the AUTH_URL defined in .cloudfuse file.

      Thanks,
      Sandeep

  21. Will Sinclair
    July 19, 2011 at 7:16 pm | #32

    Yeh, i checked with CURL and it auth’d fine. I managed to get it to work after recompiling with the london api server hardcoded instead of the default US one (didn;t have the time to debug and fix properly sorry)

    Well work is not entirely right; “cloudfuse /mountpoint” works, but then when doing any type of IO on “/mountpoint” causes the system to hang.

    Have tried rebooting a couple of times but no luck. Any ideas?

    uname -a:
    Linux 123.456.789.000 2.6.35.4-rscloud #8 SMP Mon Sep 20 15:54:33 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

    • July 19, 2011 at 7:39 pm | #33

      good that it authenticated, although me and many didn’t had to compile it with auth_url hardcoded, but I guess whatever works ;)

      regarding hanging, what steps are you doing, are you trying to copy/download a large file, bcoz it just hangs until the file download and upload finishes. don’t now exactly what could be causing it, try doing the strace on the process and see which files it’s trying to open, etc., or may be it’s problem with cloud files, the other day I was doing a large files download using swift tool and it would just hands randomly, ended up doing it with cyberduck…

      also, you can try downloading the fresh code from https://github.com/redbo/cloudfuse and compiling that, there might be any improvements. I also need to download the new code and update this whole article and share new .tar.gz but couldn’t find the time lately

      Hope it works for you, happy debugging ;)

      Cheers!
      Sandeep

  22. December 1, 2011 at 4:27 am | #34

    Thank you so much for posting this. In conjunction with other research I used a variation of your guide to compile the setup I am using today. http://noconformity.co/2011/11/30/rackspace-cloud-ubuntu-server-using-cloud-files/ Added few things like auto-mount and a direct pull from github.

    Happy Holidays.

  23. Marcel
    February 22, 2012 at 5:28 pm | #35

    Anybody have any ideas on why cloudfused containers only report back 10,000 items in a bucket? I am hoping to use cloudfuse to store lots (upto half a million) website images statically, but will need to do things like list the images. So far everything else works well using advice from http://derekarnold.tumblr.com/post/526900310/using-cloudfuse-to-mount-the-rackspace-cloud-files , and of course Sandeep.

    Surely cloudfuse uses the API and bypasses the usual restrictions on reported files inside a container? (is there an option when mounting to open the reported file quantity limits?)

    Many thanks

  24. April 4, 2012 at 5:33 am | #37

    I am having a little trouble with mounting my CloudFiles container in my CentOS 6 cloud server. Installing cloudfuse completed without any errors, mounting the storage works without any errors too. If I do an “ls” on the mount point I get the error

    ls: reading directory /mnt/cloudfiles/: Link has been severed

    Subsequent “ls” commands just freeze the console session. Something is amiss, any ideas?

    • April 22, 2012 at 8:36 pm | #38

      Hi ausip,

      Did u manage to solve the problem or still having issues with it?. To be frank I don’t have a straight answer about why it’s behaving that way. It could be that fuse driver is not loaded or it’s failing to connect to your cloud files account in which case just check that you are using correct username and API. If still no luck, then try re-building the whole thing using different revision of the code from github. Let me know how it goes.


      Sandeep

  25. 007g3m1n1
    May 14, 2012 at 3:28 am | #39

    Hi Sandeep and ausip,

    I am getting the same “Link has been severed” error on ls when the setting use_snet=true is in place inside the .cloudfuse config file.

    when that setting is removed, all works well…

    Gemini

  26. May 14, 2012 at 3:34 am | #40

    Hi Sandeep and ausip,

    I get the same “Link has been severed” error when doing an ls only when the setting use_snet=true is in my .cloudfuse config file.

    current workaround is to remove that setting

  1. April 22, 2011 at 2:41 pm | #1
  2. June 3, 2011 at 8:20 pm | #2
  3. December 1, 2011 at 4:14 am | #3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.