Mounting Rackspace Cloud Files using cloudfuse into ubuntu 10.10 v2
This article shows how to mount cloud files using cloudfuse software into your ubuntu 10.10 as a directory so you can access your cloud files containers data inside your linux server just like any other folder and files. One heads up, this gives you an easy access to your cloud files data but in no way means you can use it as a place for any database/application directly running from it, will be darn slow. So, why would one want it then? Well, there are plenty of uses of having system level access to your cloud files, for instance if you have some scripts which create your mysql backup or website backup, those scripts can create backups automatically into cloud files without you needing to copy them yourself. So, lets get on with it..
Note: The following commands are tried and tested on Ubuntu 10.10, but they should be easily applicable to other versions of ubuntu or debian. As long as you install the required packages, you should be able to compile cloudfuse code and use it.
Installing cloud fuse:
First download the cloudfuse code from cloudfuse-0.1.tar.gz. I downloaded this code from upsteam maintainer and shared it here so that all the below steps work fine, as the new code changes in repository can cause compile time errors. So, download the code from above link . Extract this file and then compile.
ssidhu@ssidhu:~$ tar -xzvf cloudfuse-0.1.tar.gz
Once you have extracted the .tar.gz file, you should have following files under cloudfuse-0.1 directory.
root@ubuntu-test:~/cloudfuse-0.1# ls -la total 280 drwxr-xr-x 3 root root 4096 Feb 21 21:47 . drwx------ 4 root root 4096 Feb 21 21:47 .. drwxr-xr-x 8 root root 4096 Feb 21 21:47 .git -rw-r--r-- 1 root root 1059 Feb 21 21:47 LICENSE -rw-r--r-- 1 root root 1024 Feb 21 21:47 Makefile.in -rw-r--r-- 1 root root 2332 Feb 21 21:47 README -rw-r--r-- 1 root root 12014 Feb 21 21:47 cloudfsapi.c -rw-r--r-- 1 root root 1043 Feb 21 21:47 cloudfsapi.h -rw-r--r-- 1 root root 11240 Feb 21 21:47 cloudfuse.c -rw-r--r-- 1 root root 4335 Feb 21 21:47 config.h.in -rwxr-xr-x 1 root root 198521 Feb 21 21:47 configure -rw-r--r-- 1 root root 1324 Feb 21 21:47 configure.in -rwxr-xr-x 1 root root 13184 Feb 21 21:47 install-sh root@ubuntu-test:~/cloudfuse-0.1#
Now its time to compile it and install it. You’ll need libcurl, libfuse, and libxml2 and their dev packages installed to build it.
Cloudfuse is built and installed like any other autoconf-configured code. Normally,
./configure make sudo make install
But, first you need to install the required packages, otherwise the ./configure command will fail and throw you errors.
apt-get update apt-get install gcc apt-get install libcurl4-openssl-dev apt-get install libxml2 libxml2-dev apt-get install libfuse-dev
now run the following command in the cloudfuse directory
root@ubuntu-test:~/cloudfuse-0.1# ./configure checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for a BSD-compatible install... /usr/bin/install -c checking for a thread-safe mkdir -p... /bin/mkdir -p checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for XML... yes checking for CURL... yes checking for FUSE... yes ......... ............ configure: creating ./config.status config.status: creating Makefile config.status: creating config.h root@ubuntu-test:~/cloudfuse-0.1# make gcc -g -O2 -I/usr/include/libxml2 -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -o cloudfuse cloudfsapi.c cloudfuse.c -lxml2 -lcurl -pthread -lfuse -lrt -ldl root@ubuntu-test:~/cloudfuse-0.1# make install /usr/bin/install -c cloudfuse /usr/local/bin/cloudfuse root@ubuntu-test:~/cloudfuse-0.1#
If everything went fine, you should have cloudfuse installed properly. Confirm this by running the which command. It should show the location of the cloudfuse binary file.
root@ubuntu-test:~/cloudfuse-0.1# which cloudfuse /usr/local/bin/cloudfuse root@ubuntu-test:~/cloudfuse-0.1#
Mounting cloudfiles:
Let’s now use cloudfuse and mount our cloudfiles.
You’ll have to create a configuration file for cloudfuse in your home directory and put your Rackspace cloudfiles username and API key in it, like below:
$HOME/.cloudfuse
username=[username]
api_key=[api key]
authurl=[auth URL]
Auth URLs:
US cloudfiles account: https://auth.api.rackspacecloud.com/v1.0
UK cloudfiles account: https://lon.auth.api.rackspacecloud.com/v1.0
The following entries are optional, you can define these values in the .cloudfuse file.
use_snet=[True to use snet for connections]
cache_timeout=[seconds for directory caching, default 600]
After creating the above configuration file, you will run the cloudfuse command like following. The syntax should be as simple as:
cloudfuse [mount point]
So, you should be able to mount cloud like this
root@ubuntu-test:/# mkdir cloudfiles root@ubuntu-test:/# cloudfuse /cloudfiles
If you run # ls -la command inside the /cloudfiles directory you should see your cloudfiles containers.
If you are not the root of the system, then you username will need to be part of “fuse” group. This can probably be accomplished with:
sudo usermod -a -G fuse [username]
If you are unable to see any containers inside the mountpoint, then probably some of the above steps didn’t work properly. You need to check and make sure that all the above steps get completed properly.
UPDATE: 30/9/2011
Here is some extra info for CentOS on how to mount cloudfuse using another use ie. Apache.
$ yum install fuse $ usermod -a -G fuse apache $ mkdir /mnt/cloudfiles $ chown apache:apache /mnt/cloudfiles $ sudo -u apache cloudfuse /mnt/cloudfiles -o username=myuser,api_key=mykey,use_snet=true,authurl="https://lon.auth.api.rackspacecloud.com/v1.0"
Play around with it and fix it how you like, but I think it would be useful. Courtesy my frnd Anh.
Let me know if there are any errors in these instructions or you faced some difficulty understanding them, I will update them accordingly. Any comments are highly appreciated.
Note: Please don’t bug Rackspace Support for the help on this article, it’s not supported by them hence this article.
Good luck!
yes this worked perfectly fine for me, i used the authurl for US cloudfiles and now i can mount my cloud files on the server
Thanks a lot sandeep for helping me out
Hi Sandeep, thanks for your previous replies. Sorry have not had time to investigate yet, but saw you added Auth URLs. It’s looking promising. Many thanks, F.
Sandeep, thanks so much. I’ve been trying to map files to my Cloud Server for a month! Just so you know, I received a link to this article from Rackspace support — they like this a lot.
Hi Trenton,
I’m glad that it worked for you, thanks for the positive feedback!
-Sandeep
Hi Sandeep,
I have mounted the cloud file to cloud server(Linux) but I acnnot copy files and folders from the cloud server to cloud file.
I got follwoing error messages:
Operation not permitted
Thanks
Hamid
Hi Sandeep, so i build a new server and tried all the steps mentioned in here and i get this error, so i guess this needs to be added to document??
$ sudo mount /media/cloudfiles
mount: can’t find /media/cloudfiles in /etc/fstab or /etc/mtab
Thanks once again for being proactive and putting up this article
Hi Mandm,
You are running the wrong command to mount the cloudfiles. do following as mentioned in the article. Cloud files cannot be mounted using normal mount command, you need to run the “cloudfuse” command as following:
mkdir /cloudfiles
cloudfuse /cloudfiles
hope it works for you.
Thanks,
Sandeep
Hi Sandeep,
Thanks for pointing out, that works like a charm. So may be that might have been the reason for the segfault issues i had where the servers would drop the mount point and also that there was a lag between both the servers seeing files uploaded from the other one…..
i’ll keep you posted on what i see
actually sandeep let me take that back, i might have mistyped the issue earlier, i use this command to load my cloud files, since i wanted to read and write to the files
sudo cloudfuse /media/cloudfiles -o allow_other,nonempty
so it still drops the mount point periodically and there is a lag when one server sees the files uploaded from other server by about 5 mins, any ideas?
hi Sandeep,
So even on the new server it gives me this error
kernel: [87106.412129] cloudfuse[19158]: segfault at 1b1 ip 00007f5d78492827 sp 00007f5d73e10558 error 4 in libxml2.so.2.7.6[7f5d783e8000+146000]
So what verion of libxml2 so you have?
mine is
$ sudo dpkg –list libxml2
Version Description
+++-=====================-======================-==================================
ii libxml2 2.7.6.dfsg-1ubuntu1.1 GNOME XML library
This works great for me, but I can not get it to allow anyone but root to write to this location. Is there a way to mount it read-write for a non-root user? The mount command claims it is mounted rw, but it won’t let other users cd to the directory or copy files there.
hi Jody, you should use this command
sudo cloudfuse /media/cloudfiles -o allow_other,nonempty
this will help you read write files other than the root user, even thouugh the file ownership and group ownership shows as root root, it is stil accessible via apache user..etc
That worked, thank you. Now I just need to get automount with /etc/fstab working. It won’t auto-mount at boot. If anyone can help, I am using the following line in that file:
cloudfuse /home/user/cloud fuse defaults,allow_other,gid=1002,umask=007,username=cloudusername,api_key=XXXXXXXXX 0 0
Also, if I have that in fstab and I try mount -a I get this output:
mount: wrong fs type, bad option, bad superblock on cloudfuse,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog – try
dmesg | tail or so
I think a clue is that after a reboot, if I manually try to mount it, I initially get this message:
fuse: device not found, try ‘modprobe fuse’ first
and if I enter ‘modprobe fuse’ and then try to manually mount, it works.
Can anyone tell me what I need to do to make this work with fstab?
Yeah…same issues that i faced…:)
so i don’t use the /etc/fstab, for mounting the cloudfiles on server reboot try this
add these entries to the /etc/crontab
@reboot root modprobe fuse
@reboot root cloudfuse /media/cloudfiles -o allow_other,nonempty
and try restarting the server, you should see the cloudfiles mounted
thanks mandm, this is really great and makes me so much happy, we all helping each other, that’s what open source is all about..
I’ll go ahead add these instructions to the post as well. once again thanks for helping out Jody.
I noticed a oddity with cron under CentOS not setting the $HOME variable for the executing user, so the cloudfiles account never gets mounted as it can’t find the $HOME/.cloudfuse file. To get around this we can explicitly specify the $HOME variable to use at runtime,
@reboot root HOME=/root cloudfuse /media/cloudfiles -o allow_other,nonempty
Alternative, it is also possible to specify all mount options in the crontab.
Not a problem Sandeep, i am just trying to share as i learn..:)
That worked perfectly! Thanks mandm!
Hi Sandeep,
is there a way to get around the 10000 files limit per container ?
I have hit the limit and the wierd part is that any file or folder created by my application gets deleted if it has letters greater than ‘f’ since i have 70K files which starts with ‘f’
I have twice tried to use this to sync files as a backup mechanism, and twice it (or something related) has corrupted the local filesystem.
# ll /mnt
ls: cannot access /mnt/cloud: Transport endpoint is not connected
total 8.0K
drwxr-xr-x 4 root root 4.0K 2011-05-18 10:23 ./
drwxr-xr-x 24 root root 4.0K 2011-05-16 13:54 ../
d????????? ? ? ? ? ? cloud/
drwxr-xr-x 2 root root 0 1969-12-31 19:00 rackspace/
The rackspace mount point does not look too screwy, but any attempt to access it results in a hung terminal. The cloud one is obviously fubar.
Yes i agree it is a complete fubar in terms of reliability and scalability. You can only use this in a non productive and an environment where the users have no interaction.
I have a cronjob running every minute to get around this error and once the jobs sees that the transport endpoint is not connected it umounts and remounts the drive
But yes, there is no point in using it for scalable and highly visible environments
(Using it since Jan 2011)
I am having better results just mounting on demand. I am using this for backups and this is done nightly. So I unmount as soon as the backup is completed. That is helping. I have also identified some other “issues” that have caused reliability problems. Copying 2G+ files (at least from a 32bit system) will cause a hang. The same files move easily between local Linux 32bit systems, and to Cloud Servers. And in fact with other tools to Cloud files. I have also found 2 files with very strange character encoding in the filenames, that would cause a dropped connection. There may be some rsync options too that don’t play nice.
Hi Sandeep,
Good Work,
I am new to Linux. trying hard to work with this.. can you guide me few steps to learn Linux as early as possible.
Hi – when I try to mount, I get the error:
root@host [/]# cloudfuse /cloudfiles
Unable to authenticate.
I’ve verified the API and username with Rackspace and all are correct. Any idea what could be wrong?
Brad, make sure you are using correct API AUTH URL, US and UK cloud files accounts has different URLs. Also, are you using the code which I have shared in the blog or you checked out directly from github? if you checked out directly from github something might have changed in the code. The source code which I have shared with the link in the article work fine with all the instructions, so it might be AUTH URL or make sure your linux has ca-bunble certs installed, test the authentication with cURL first as well.
good luck!
Sandeep
Hi Hal… What are you using to get around that “Transport endpoint…” error to unmount the drive? I have the same error and wanted to take your recommendations to mount/unmount on demand but I don’t see any unmount options within cloudfuse?
-st
Sandeep,
I am also having the
“Unable to authenticate.
Bad username or password” error
I have followed your instructions meticulously, and triple checked everything is as you say it should be.
How do I check if my server has ca-bunble?
I worked out the issue, in cloudfuse.c you are hardcoding the apiurl option and ignoring the one pulled from ~/.cloudfuse … I am auth’ing against the london server.
Cheers,
W
Well i can mount fine, but when trying to access the mountpoint it just hangs, any ideas?
Something similar happens to me. Occasionally my cloudfuse mount will just hang if I try to access it from a shell. If I umount it, I get an error stating it’s busy. This has happened twice in the last couple months. A reboot of the system solved it.
Hi Will,
check with curl, make sure you can authenticate to the AUTH_URL with https and your curl shouldn’t throw any error. If the curl works fine that means ca-bundle is working fine.
http://www.productionscale.com/home/2009/8/2/using-curl-to-access-the-rackspace-cloud-api.html#axzz1SZyo778r
and regarding AUTH_URL, did you check out the new code from github or used the one which I’m sharing in this post, the code which I’m sharing works fine with all the instructions on this page and it does check the .cloudfuse configuration and use the AUTH_URL defined in .cloudfuse file.
Thanks,
Sandeep
Yeh, i checked with CURL and it auth’d fine. I managed to get it to work after recompiling with the london api server hardcoded instead of the default US one (didn;t have the time to debug and fix properly sorry)
Well work is not entirely right; “cloudfuse /mountpoint” works, but then when doing any type of IO on “/mountpoint” causes the system to hang.
Have tried rebooting a couple of times but no luck. Any ideas?
uname -a:
Linux 123.456.789.000 2.6.35.4-rscloud #8 SMP Mon Sep 20 15:54:33 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
good that it authenticated, although me and many didn’t had to compile it with auth_url hardcoded, but I guess whatever works
regarding hanging, what steps are you doing, are you trying to copy/download a large file, bcoz it just hangs until the file download and upload finishes. don’t now exactly what could be causing it, try doing the strace on the process and see which files it’s trying to open, etc., or may be it’s problem with cloud files, the other day I was doing a large files download using swift tool and it would just hands randomly, ended up doing it with cyberduck…
also, you can try downloading the fresh code from https://github.com/redbo/cloudfuse and compiling that, there might be any improvements. I also need to download the new code and update this whole article and share new .tar.gz but couldn’t find the time lately
Hope it works for you, happy debugging
Cheers!
Sandeep
Thank you so much for posting this. In conjunction with other research I used a variation of your guide to compile the setup I am using today. http://noconformity.co/2011/11/30/rackspace-cloud-ubuntu-server-using-cloud-files/ Added few things like auto-mount and a direct pull from github.
Happy Holidays.
Anybody have any ideas on why cloudfused containers only report back 10,000 items in a bucket? I am hoping to use cloudfuse to store lots (upto half a million) website images statically, but will need to do things like list the images. So far everything else works well using advice from http://derekarnold.tumblr.com/post/526900310/using-cloudfuse-to-mount-the-rackspace-cloud-files , and of course Sandeep.
Surely cloudfuse uses the API and bypasses the usual restrictions on reported files inside a container? (is there an option when mounting to open the reported file quantity limits?)
Many thanks
Hi Marcel,
Cloud Files API itself provides a way to list files more than 10K. The second listing needs to be requested by providing marker as outlined in API docs.
http://docs.rackspace.com/files/api/v1/cf-devguide/content/List_Large_Number_of_Objects-d1e1521.html
Cloudfuse software itself does not implement this functionality, it simply requests the listing of a directory once and then shows that, does not request any additional listing with a marker. If you are a C developer may be you can extend redbo’s code and include that functionality.
Looks like somebody already did this.
https://github.com/redbo/cloudfuse/pull/24
If you check out code from this commit from and build it again, it should work fine.
https://github.com/dsnopek/cloudfuse/commit/db7be74b3da460496b5d3d2b612753a788d80d52
Hope it helps.
Note: All this software is open source and is not from me, I have only outlined the steps to use it. So, please use the above and other instructions on your own discretion.
Cheers,
Sandeep
I am having a little trouble with mounting my CloudFiles container in my CentOS 6 cloud server. Installing cloudfuse completed without any errors, mounting the storage works without any errors too. If I do an “ls” on the mount point I get the error
ls: reading directory /mnt/cloudfiles/: Link has been severed
Subsequent “ls” commands just freeze the console session. Something is amiss, any ideas?
Hi ausip,
Did u manage to solve the problem or still having issues with it?. To be frank I don’t have a straight answer about why it’s behaving that way. It could be that fuse driver is not loaded or it’s failing to connect to your cloud files account in which case just check that you are using correct username and API. If still no luck, then try re-building the whole thing using different revision of the code from github. Let me know how it goes.
–
Sandeep
Hi Sandeep and ausip,
I am getting the same “Link has been severed” error on ls when the setting use_snet=true is in place inside the .cloudfuse config file.
when that setting is removed, all works well…
Gemini
Hi Sandeep and ausip,
I get the same “Link has been severed” error when doing an ls only when the setting use_snet=true is in my .cloudfuse config file.
current workaround is to remove that setting