Academic project Fall 2019 (Distributed Systems Course)
Please create a folder say "namenode" and inside this folder create a file namenode_hosts.conf. Configuration of this file are shown below
PORT
Here PORT is the number of port that the namenode will be using to listen to new datanodes/clients.
Please create a folder say "datanode" and inside this folder create a file datanode_hosts.conf. Configuration is shown below
DataNodeIP DataNodePORT NameNodeIP NameNodePORT DataNodeName
DataNodeIP: is the IP of the machine DataNodePORT: is the port which the datanode will listen to other nodes. NameNodeIP: is the name node ip NameNodePORT: is the port which the namenode will listen to other nodes. DataNodeName: any string would do.
Create a folder called client containing one file called client_hosts.conf having the following configuration
ClientIP NameNodeIP NameNodePORT ClientName
ClientIP: is the IP of your current machine NameNodeIP: is the name node ip NameNodePORT: is the port which the namenode will listen to other nodes. ClientName: any string would do.
Please Note if you are using AWS instances or you are using public ips. You have to permit the ports to access to all TCP. Since we are using random ports for connecting the nodes better have all the range of the ports connect to all TCP.
On namenode run:
sudo docker run -v $(pwd):/namenode -it --network host jafarbadour/namenode-dfs:latest
you have also to configure users.conf which is a json file of {user:password} example
{"hussain":"123456","almir":"123456","jafar":"123456"}
Make sure there is only one line in the JSON file.
On datanode run:
sudo docker run -v $(pwd):/datanode -it --network host -d jafarbadour/datanode-dfs:latest
On clientnode run:
sudo docker run -v $(pwd):/client -it --network host jafarbadour/client-dfs:latest
-v will let docker container mount the folder (namenode,datanode,client) in the host OS file system
When initialization of namenode in 3rd step in instance configuration -> advanced details -> User data write the following script
#!/bin/bash
snap install docker
mkdir /home/ubuntu/namenode
echo "26200" >>/home/ubuntu/namenode/namenode_hosts.conf
echo "{\"hussain\":\"123456\",\"almir\":\"123456\",\"jafar\":\"123456\"}" >>/home/ubuntu/namenode/users.conf
Initialization of datanodes
#!/bin/bash
snap install docker
mkdir /home/ubuntu/datanode
echo "25011 18.217.199.246 26200 datanode" >>/home/ubuntu/datanode/datanode_hosts.conf
cd /home/ubuntu/datanode
sudo docker pull jafarbadour/datanode-dfs:latest
sudo docker run -v $(pwd):/datanode -it -d --network host jafarbadour/datanode-dfs:latest
where in the last line we have the ip:port of the namenode as mentioned above in the datanode_hosts.conf
Initialization of clientnode
#!/bin/bash
snap install docker
mkdir /home/ubuntu/client
echo "18.217.199.246 26200 client" >>/home/ubuntu/client/client_hosts.conf
login
login hussain 123456
format
Just write format
create
create /a/b/c/f.txt
upload pathInClient pathOnServer
upload ./clientfiles/toupload.pdf ./downloads/book.pdf
download pathOnClient pathOnServer
download ./downloads/book.pdf ./serverbooks/book.pdf
delete pathToFileOnServer
delete /a/b/c/xx.yy
info pathToFileOnServer
info /a/b/c/xx.yy
cp firstFilePathOnServer nameOfFileInNewPath
cp /a/b/c/book.pdf /d/e/f/book-copy.pdf
mv firstFilePathOnServer nameOfFileInNewPath
mv /a/b/c/book.pdf /d/e/f/book-moved.pdf
cd changeDirectory
cd /newdir/newsubdir1/
ls listAllFiles/Directories
ls
ls /a/
ls .
ls ..
mkdir createNewDirectory
mkdir newdir
mkdir newdir/newdir2/
rmdir pathToDir -r // Force recursive deletion
rmdir /a/b/c/ (must be empty dir)
rmdir /a/b/c/ -r (doesn't matter)
when one wants to build the docker images and push to docker hub please follow this commands
chmod +x runMeTodockerize.sh
./runMeTodockerize.sh
