Build Your Own Cloud Stor­age Sys­tem Us­ing OSS

The real threats to stored data are breaches which, of late, have been af­fect­ing many cloud ser­vice providers. Se­cu­rity vul­ner­a­bil­i­ties that en­able breaches re­sult in a loss of mil­lions of user cre­den­tials. In this ar­ti­cle, we ex­plore the prospects of set

OpenSource For You - - Contents - By: Swap­neel Me­hta The au­thor has worked with Mi­crosoft Re­search, CERN and star­tups in AI and cy­ber se­cu­rity. An open source en­thu­si­ast, he en­joys spend­ing his time or­gan­is­ing soft­ware de­vel­op­ment work­shops for school and col­lege stu­dents. You can con­nect

The Euro­pean Or­gan­i­sa­tion for Nuclear Re­search (CERN), a re­search col­lab­o­ra­tion of over 20 coun­tries, has a unique prob­lem—it has way more data than it is pos­si­ble to store! We’re talk­ing about petabytes of data per year, where one petabyte equals a mil­lion gi­ga­bytes. There are en­tire de­part­ments of sci­en­tists work­ing on a sub­ject termed DAQ (Data Ac­qui­si­tion and Fil­ter­ing), sim­ply to fil­ter out 95 per cent of the ex­per­i­ment­gen­er­ated data and store only the use­ful 5 per cent. In fact, it has been es­ti­mated that data in the dig­i­tal uni­verse will amount to 40 zettabytes by 2020, which is about 5,000 gi­ga­bytes of data per per­son.

With the re­cent spate of breaches af­fect­ing cloud ser­vice providers, set­ting up a per­sonal data store or even a pri­vate cloud be­comes an at­trac­tive prospect.

Data stor­age in­fra­struc­ture is broadly clas­si­fied as ob­ject-based, block stor­age and file sys­tems, each with its own set of fea­tures.

Ob­ject-based stor­age

This con­struct man­ages data as ob­jects in­stead of treat­ing it as a hi­er­ar­chy of files or blocks. Each ob­ject is as­so­ci­ated with a unique iden­ti­fier and com­prises not only the data but also, in some cases, the meta­data. This stor­age pat­tern seeks to en­able ca­pa­bil­i­ties such as ap­pli­ca­tion pro­gram­mable in­ter­faces, data man­age­ment such as repli­ca­tion at ob­jectscale, etc. It is of­ten used to al­low for the re­ten­tion of mas­sive amounts of data. Ex­am­ples in­clude the stor­age of pho­tos, songs and files on a mas­sive scale by Facebook, Spo­tify and Drop­box, re­spec­tively.

Block stor­age

Data is stored as a se­quence of bytes, termed a phys­i­cal record. This so called ‘block’ of data com­prises a whole num­ber of records. The process of putting data into blocks is termed as block­ing, while the re­verse is called de­block­ing. Block­ing is widely em­ployed when stor­ing data to cer­tain types of mag­netic tape, Flash me­mory and ro­tat­ing me­dia.

File sys­tems

These data stor­age struc­tures fol­low a hi­er­ar­chy, which con­trols how data is stored and retrieved. In the ab­sence of a file sys­tem, in­for­ma­tion would sim­ply be a large body of data with no way to iso­late in­di­vid­ual pieces of in­for­ma­tion from the whole. A file sys­tem en­cap­su­lates the com­plete set of rules and logic used to man­age sets of data. File sys­tems can be used on a va­ri­ety of stor­age me­dia, most com­monly, hard disk drives (HDDs), mag­netic tapes and op­ti­cal discs.

Build­ing open source stor­age Soft­ware

Net­work At­tached Stor­age (NAS) pro­vides a sta­ble and widely em­ployed al­ter­na­tive for data stor­age and shar­ing across a net­work. It pro­vides cen­tralised repos­i­tory of data that can be ac­cessed by dif­fer­ent mem­bers within the or­gan­i­sa­tion. Vari­a­tions in­clude pro­vid­ing com­plete soft­ware and hard­ware pack­ages serv­ing as out-of-the-box al­ter­na­tives. These in­clude soft­ware and file sys­tems such as Glus­ter, Ceph, NAS4Free, FreeNAS, and others. As an ex­am­ple, we will look into the gen­eral steps in­volved in de­ploy­ing such a sys­tem by tak­ing the case of a pop­u­lar rep­re­sen­ta­tive of the set.


With en­ter­prise-grade fea­tures, richly sup­ported plug­ins, and an en­ter­prise-ready ZFS file sys­tem, it is easy to see why FreeNAS is one of the most pop­u­lar op­er­at­ing sys­tems in the mar­ket for data stor­age.

Let’s take a deeper look at file sys­tems since they are widely used in set­ting up stor­age net­works to­day. Build­ing your own data stor­age us­ing FreeNAS in­volves fol­low­ing a few of the fol­low­ing simple steps:

1. You will need to down­load the disk im­age suit­able for your ar­chi­tec­ture and burn it onto ei­ther a USB stick or a CD-ROM, as per your pref­er­ence.

2. Since you will be boot­ing your new disk or ma­chine with FreeNAS, you will need to open the BIOS set­tings on boot­ing it, and set the boot pref­er­ence to USB so that your sys­tem first tries to boot from the USB and, if not found, then from other at­tached me­dia.

3. Once you have cre­ated the stor­age me­dia with the re­quired soft­ware, you can boot up your sys­tem and in­stall FreeNAS in the des­ig­nated par­ti­tion.

4. Hav­ing set the root pass­word, when you boot into it af­ter in­stal­la­tion, you will have the op­tion of us­ing the Web GUI to log into the sys­tem. For some users, it might be much more in­tu­itive to use this op­tion as com­pared to the con­sole-based lo­gin.

5. Us­ing the GUI or con­sole, you can con­fig­ure and man­age

your stor­age op­tions de­pend­ing on your ap­pli­ca­tion(s).

Pri­vate cloud stor­age

An­other re­cent trend is cloud stor­age, given the sud­den reduction in free cloud stor­age of­fered by providers like Mi­crosoft and Drop­box. Pub­lic clouds have multi-ten­ancy in­fra­struc­ture and al­low for great scal­a­bil­ity and flex­i­bil­ity, ab­stract­ing away the com­plex­i­ties as­so­ci­ated with de­ploy­ing and main­tain­ing hard­ware. For in­stance, the cre­ators of Glus­ter re­cently came out with an open source pro­ject called Minio to pro­vide this func­tion­al­ity to users. One of the ser­vices we will look at is ownCloud, a Drop­box al­ter­na­tive, that of­fers sim­i­lar func­tion­al­ity, along with the ad­van­tage of be­ing open source.

1. In or­der to build a pri­vate cloud, you re­quire a server run­ning an op­er­at­ing sys­tem such as Linux or Win­dows. ownCloud al­lows clients to be in­stalled on such a

Linux server. 2. While in­stalling and run­ning an Apache server on Linux, the up­load­_­max_­file­size and post_­max_­file­size flags need to be up­dated to higher val­ues than the de­fault (2MB).

3. The sys­tem is re­quired to have MySQL, PHP (5.4+), Apache, GD and cURL in­stalled be­fore pro­ceed­ing with the ownCloud in­stal­la­tion. Fur­ther, a data­base must be cre­ated with priv­i­leges granted to a new user.

4. Once the sys­tem is set up, pro­ceed with down­load­ing the ownCloud files and ex­tract them to /var/www/ownCloud. 5. Change the Apache virtual host to point to this ownCloud di­rec­tory by mod­i­fy­ing the doc­u­ment root in /etc/apache2/ sites­avail­able/000­de­fault.conf to /var/www/ownCloud. 6. Fi­nally, type in the IP ad­dress of the server in your browser and you should be able to ar­rive at the lo­gin screen.

While there are trade-offs be­tween cloud-based stor­age and tra­di­tional means of stor­age, the for­mer is a highly flex­i­ble, sim­pli­fied and se­cure model of data stor­age. And with the providers of­fer­ing more con­trol over de­ploy­ments, pri­vate clouds may well be the main file stor­age op­tions in the near fu­ture!

Fig­ure 3: Se­lect­ing the boot par­ti­tion [Source:]

Fig­ure 1: Hacked cre­den­tials

Fig­ure 4: FreeNAS GUI [Source:]

Fig­ure 2: Ob­ject stor­age, file sys­tems and block stor­age [Source:]

Im­age 8: Fi­nal con­fig­u­ra­tion for ownCloud [Source: it­tuto­ri­]

Fig­ure 7: Edit­ing the doc­u­ment root in the con­fig­u­ra­tion files [Source: it­tuto­ri­]

Fig­ure 5: Con­fig­ur­ing stor­age op­tions [Source:]

Fig­ure 6: Set­ting up a pri­vate cloud [Source:]

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.