How to build a Beowulf Cluster Applied to Computational

Сomentários

Transcrição

How to build a Beowulf Cluster Applied to Computational
How to build a Beowulf Cluster Applied to
Computational Electromagnetic
Carlos Henrique da Silva Santos
Leonardo André Ambrosio
Hugo Enrique Hernández Figueroa
Department of Microwaves and Optics (DMO)
School of Electrical and Computer Engineering (FEEC)
State University of Campinas (UNICAMP)
Summary
•
•
1.
2.
3.
4.
•
•
Motivation;
Objectives;
Preview: Operation Systems and Network concepts;
Installing Linux Fedora distribution;
Configuring a computer cluster;
HPC cluster applications in electromagnetism;
Conclusions;
Main references;
How to Build a Cluster
Carlos Henrique
2
Motivations
• Different concepts over High Performance Computing (HPC) applied to
Computational Electromagnetism;
• Low computational costs solutions for HPC;
• New computational technologies and resources to build a HPC cluster;
• Some large computational electromagnetic application.
How to Build a Cluster
Carlos Henrique
3
Objectives
• Presenting some computational concepts:
• Operating system layers
• Network structures;
• Linux distributions and services (NFS and SSH);
• Sophisticated and easy ways to build a computer cluster, focusing on data
security;
• How to configure services and connections.
How to Build a Cluster
Carlos Henrique
4
1st Part - Concepts
Operating System
Linux
Network
How to Build a Cluster
Carlos Henrique
5
Operating Systems
Kernel
Hardware
Kernel
System Calls
Program
How to Build a Cluster
Carlos Henrique
6
Operating System: Structure
•
Memory management;
•
Disk and file systems;
•
Networking;
•
Security:
• Internal: Programs running locally;
• External: Network connections;
•
Graphical user interface;
•
Device Driver;
•
Process management.
How to Build a Cluster
Firewall
Carlos Henrique
7
Operating System: Process
• Process:
• A program in execution;
• An instance of a program running in a computer;
• The entity that can be assigned and executed on a processor (program
code and a dataset);
Identifier
State
Priority
Program counter
Memory pointers
Context data
I/O Status information
Accounting information
.
.
.
How to Build a Cluster
Carlos Henrique
8
Modern Unix System
coff
a.out
elf
NFS
Exec
switch
File
mappings
FFS
virtual
memory
framework
Device
mappings
vnode/vfs
interface
Anonymous
mappings
RFS
Common
facilities
Disk driver
Time sharing
process
block
device
switch
Scheduler
framework
Tape driver
System
processes
Streams
Network
driver
How to Build a Cluster
s5fs
Carlos Henrique
tty
driver
9
Why Linux is necessary?
• Linux is a free operating system;
• It was concepted by Linus Torvalds to attempt his necessities, without
intentions to get money;
• The development style adopted is the collective help, where someone
manages the colective efforts of a group to improve the system;
• Millions of people are gratuitously contributing to develop Linux, for pleasure
or to have a better operating system.
How to Build a Cluster
Carlos Henrique
10
Operating System: Linux
• It is a multiprocessor and multiuser system, which means that many users
have the possibility to execute many processes into the same computer using
Linux in the same time;
• Free and open source software, basing on the GPL. Furthermore, most of the
computational tools for this operational system are free too;
• It has some different Kernel to provide better solutions depending on the
application;
• It is one of the best operational system for network and HPC applications;
• Free tutorial, manual and help on the Internet.
How to Build a Cluster
Carlos Henrique
11
Networking Application: Computer Cluster
0
How to Build a Cluster
Carlos Henrique
1
12
Network: Services, Protocols and Layers
• Protocols: are related to the packets sent between peer entities on different
machines;
• Services: is a set of primitives (operations) that a layer provides to the layer
above it. In other words, is related to the interface between layers.
Machine 1
Machine 2
Layer k+1
Layer k+1
Service provided by layer k
Protocol
Layer k
Layer k-1
Layer k-1
How to Build a Cluster
Layer k
Carlos Henrique
13
Network: OSI Reference Model layers
7
Application
Application protocol
Application
APDU
Presentation
PPDU
Session
SPDU
Transport
TPDU
Interface
6
Presentation
5
Session
4
Transport
Presentation protocol
Session protocol
Transport protocol
Communication subnet boundary
Internet subnet protocol
3
Network
Network
Network
Network
Packet
2
Data link
Data link
Data link
Data link
Data link
1
Physical
Physical
Physical
Physical
Bit
How to Build a Cluster
Carlos Henrique
14
Network: Open Systems Interconnection (ISO-OSI) Reference Model
•
Deals with connecting open system, which are open for communication with
other systems;
•
OSI model has seven layers, based on:
1. A layer should be created where a different abstraction is needed;
2. Each layer should perform a well-defined function;
3. Each layer function should be defined in order to provide an
international standardization protocol;
4. The layer boundaries should be chosen to minimize the information
flow across the interfaces;
5. The number of layer should be large enough that distinct functions
and small enough that the architecture does not become widely.
How to Build a Cluster
Carlos Henrique
15
2nd Part – Installing Fedora RedHat
How to Build a Cluster
Carlos Henrique
16
Choosing installation version
• i386
• Intel x86-compatible processors, including Intel Pentium and PentiumMMX, Pentium Pro, Pentium-II, Pentium-III, Celeron, Pentium 4, Xeon,
Core Duo, and Centrino/Centrino Duo; VIA C3/C3-m and Eden/Eden-N;
and AMD Athlon, AthlonXP, Duron, AthlonMP, Sempron, and Turion
• ppc
• PowerPC processors, such as those found in Apple Power Macintosh, G3,
G4, and G5, and IBM pSeries systems
• x86_64
• 64-bit AMD processors such as Athlon64/FX/X2, Turion64, Opteron; and
Intel 64-bit processors such as EM64T
How to Build a Cluster
Carlos Henrique
17
Pre-installation
How to Build a Cluster
Carlos Henrique
18
Installation: System properties
How to Build a Cluster
Carlos Henrique
19
Installation: Operating System Resource
How to Build a Cluster
Carlos Henrique
20
3rd Part – Cluster
• Configuring Linux
• Network
• SSH
• NFS
• Installing MPI Libraries
How to Build a Cluster
Carlos Henrique
21
TCP: Transport Control Protocol
• Transport Layer at OSI Model
• It is connection oriented, which means that all problems no solved in IP
level will be solved here;
• Some problems that must be solved by TCP:
• Lost of packages or destructions occurs
transmission;
• Package expedition out of order or duplicated.
How to Build a Cluster
Carlos Henrique
during the
22
TCP: Transport Control Protocol
• The TCP specify the data package format and patterns recognized in
reliable changes between two computers, like the procedures that were
applied in computers to ensure the data communication;
• Procedures to be considered:
• Distinguishing between multiples destinies in a determined machine;
• Error recovering, like some lost packages or duplicated.
How to Build a Cluster
Carlos Henrique
23
TCP: Transport Control Protocol
• Available multiple programs runs in one machines communicate;
• Multiplex the TCP input traffic between the applications programs;
• Uses numerical port to identify the last destination in a machine. Each port
is identified by a low integer number;
• Example: 128.10.2.3.25 specifies the TCP port number 25, in the machine
with IP address 128.10.2.3.
How to Build a Cluster
Carlos Henrique
24
IP: Internet Protocol
Network Configuration - Structure
http://www.informabr.com.br/ipsubnet.htm
How to Build a Cluster
Carlos Henrique
25
Linux Network Configuration: shell Mode
• Identify network device in use;
• Open the file:
• /etc/sysconfig/network-scripts/ifcfg-ethx
DEVICE=ethx
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
IPADDR=10.0.0.100
NETWORK=10.0.0.0
BROADCAST=10.0.0.255
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
DNS1=10.0.0.1
How to Build a Cluster
#
#
#
#
#
#
#
#
#
#
Identify the device
Type of boot (none, static ou dhcp)
Starting on machine boot
Type of network
IP address of the machine at intranet
Network Masquerade
Range of IP address in the network
Defines the subnet
Package destination to network exit
Import
the
DNS
specified
on
/etc/resolv.conf
Carlos Henrique
26
Shell Script – Network Address Translation
modprobe
modprobe
modprobe
modprobe
modprobe
modprobe
modprobe
echo 1 >
echo 1 >
echo 1 >
echo 1 >
for f in
echo
done
for f in
echo
done
for f in
echo
done
for f in
echo
done
iptables
How to Build a Cluster
ip_conntrack
ip_conntrack_ftp
ip_tables
ipt_state
iptable_nat
ip_nat_ftp
ipt_limit
/proc/sys/net/ipv4/ip_forward
/proc/sys/net/ipv4/tcp_syncookies
/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
/proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
/proc/sys/net/ipv4/conf/*/rp_filter; do
1 >$f
/proc/sys/net/ipv4/conf/*/accept_redirects; do
0 >$f
/proc/sys/net/ipv4/conf/*/send_redirects; do
0 >$f
/proc/sys/net/ipv4/conf/*/accept_source_route; do
0 >$f
-t nat -A POSTROUTING -o eth0 -j MASQUERADE
Carlos Henrique
27
Main Network Services Requested
• FTP/SFTP (File Transfer Protocol/SSH File Transfer Protocol): to share
and access files in different machines or networks;
• SSH: Secure Shell: remote access control;
• NFS (Network File System): sharing files in a network
How to Build a Cluster
Carlos Henrique
28
SSH: Secure Shell
• O SSH has been defined on IANA beyond the RFC 4250;
• A free and open-source available is OpenSSH, which supports the versions
1.3, 1.5 e 2.0 of the SSH protocol;
• Shell command:
ssh cepofslave
• The first time that a connection for a different is required, the following
message is exhibit:
The authenticity of host ‘cepofslave’ can’t be established.
DSA key fingerprint is 95:90:3a:3a:bc:f3:9a:9b:01:5d:b3:07:38:e2:11:0c.
Are you sure you want to continue connecting (yes/no)?
How to Build a Cluster
Carlos Henrique
29
NFS: Network File System
• It is a model to structure data file distribution in some servers, mounting a
virtual directory.;
• Making possible to users to centralize their folders in a server;
• These tools is important in High Performance Computing structures that
access disc. Computer cluster is an evident example.
How to Build a Cluster
Carlos Henrique
30
NFS: Network File System
• To allow client access in NFS servers the following Daemons are
necessary:
• nfsd: to receive NFS clients requests;
• mountd: daemon to mount NFS partitions and to execute nfsd
requests;
• portmap: allows to NFS clients dicover which NFS server port is
using.
How to Build a Cluster
Carlos Henrique
31
NFS: Network File System
• A client can connect a shared data file, he needs of permission to do it. For
this reason, is important to certify if he is allowed in the file /etc/exports;
• In the /etc/exports each line represents one file system to be exported. A
remote system can be specify just only one time by the file system, and
only one standard input can be defined.
IP range identification
/home/cluster 10.0.0.0/255.255.255.0(rw,no_root_squash)
Identifying IPs
/home/cluster 10.0.0.100(rw,no_root_squash)
/home/cluster 10.0.0.101(rw,no_root_squash)
How to Build a Cluster
Carlos Henrique
32
Configure Cluster: Server
Creating one user called cepof
Step 0
adduser cepof
passwd cepof
/etc/exports
NFS
Step 1
/home/cepof 10.0.0.101 (rw, no_root_squash)
/home/cepof 10.0.0.102 (rw, no_root_squash)
/home/cepof 10.0.0.103 (rw, no_root_squash)
/etc/hosts
Step 3
10.0.0.101
10.0.0.102
10.0.0.103
cepofslave1
cepofslave2
cepofslave3
/etc/hosts.equiv
Step 4
How to Build a Cluster
cepofslave1
cepofslave2
cepofslave3
Carlos Henrique
33
Configure Cluster: Server
Reboot NFS service
Step 5
service nfs restart
Change to cepof user
Step 6
su cepof
Creating the cryptographic key to access
Step 7
ssh-keygen -t dsa
Creating authorization file
Step 8
cat .ssh/id_dsa.pub > authorized_keys
Permission of the directory .ssh
Step 9
chmod 700 .ssh
File access permission
Step 10
How to Build a Cluster
chmod 644 .ssh/authorized_keys
Carlos Henrique
34
Configure Cluster: Slave nodes
Creating cepof user
Step 0
adduser cepof
Passwd cepof
/etc/hosts
Step 1
Step 2
How to Build a Cluster
10.0.0.101
10.0.0.102
10.0.0.103
cepofslave1
cepofslave2
cepofslave3
/etc/fstab
10.0.0.1:/home/cluster /home/cluster nfs
Carlos Henrique
exec,dev,suid,rw 1 1
35
Configure Cluster: Slave nodes
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
How to Build a Cluster
FTP connection on the Server
sftp [email protected]
Getting the public key
get .ssh/id_dsa.pub
Ending connection
exit
Creating the authorization key file
cat .ssh/id_dsa.pub > authorized_keys
Permissions of the .ssh directory
chmod 700 .ssh
Permissions of the authorization file
chmod 644 .ssh/authorized_keys
Carlos Henrique
36
Example: “Hello World”
#include <unistd.h>
#include <stdio.h>
#include "mpi.h"
int main(int argc, char ** argv)
{
int my_rank;
int p;
int source;
int dest;
int silen = 128;
int gherr;
char hname[128];
char message[800];
MPI_Status status;
int tag = 50;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
MPI_Comm_size(MPI_COMM_WORLD, &p);
gherr = gethostname( hname, silen);
if (my_rank != 0) {
sprintf(message, "Greetings from process %d on %s!", my_rank, hname);
dest = 0;
MPI_Send(message, strlen (message)+1, MPI_CHAR, dest,
tag, MPI_COMM_WORLD);
} else {
printf ("Messages received by process %d on %s.\n\n", my_rank, hname);
for (source = 1; source < p; source++) {
MPI_Recv(message, 800, MPI_CHAR, source, tag,
MPI_COMM_WORLD, &status);
printf("%s\n", message);
}
}
MPI_Finalize();
return 0;
}
How to Build a Cluster
Carlos Henrique
37
Parallel Process Schematic
Process
Process00(Mater
(MaterNode)
Node)
Process
Process11
How to Build a Cluster
Process
Process22
Carlos Henrique
Process
Process33
38
Creating boot configuration file
• vi lamhosts
• Add every IP or hosts of the cluster machines
10.0.0.1
10.0.0.101
10.0.0.102
LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University
• lamboot –v lamhosts
• mpiCC hello.c –o hello
• mpirun C hello
How to Build a Cluster
Carlos Henrique
n-1<13229> ssi:boot:base:linear: booting n0 (10.0.0.1)
n-1<13229> ssi:boot:base:linear: booting n1 (10.0.0.102)
n-1<13229> ssi:boot:base:linear: booting n2 (10.0.0.103)
n-1<13229> ssi:boot:base:linear: booting n3 (10.0.0.104)
n-1<13229> ssi:boot:base:linear: booting n4 (10.0.0.105)
n-1<13229> ssi:boot:base:linear: booting n5 (10.0.0.106)
n-1<13229> ssi:boot:base:linear: booting n6 (10.0.0.107)
n-1<13229> ssi:boot:base:linear: booting n7 (10.0.0.108)
n-1<13229> ssi:boot:base:linear: booting n8 (10.0.0.110)
n-1<13229> ssi:boot:base:linear: booting n9 (10.0.0.111)
n-1<13229> ssi:boot:base:linear: booting n10 (10.0.0.112)
n-1<13229> ssi:boot:base:linear: booting n11 (10.0.0.113)
n-1<13229> ssi:boot:base:linear: booting n12 (10.0.0.114)
n-1<13229> ssi:boot:base:linear: finished
39
4th Part – Applications of HPC
Electromagnetic Applications
How to Build a Cluster
Carlos Henrique
40
Cluster Application: Human Head SAR Simulation
Slice 1
Slice 2
Slice 4
y
x
How to Build a Cluster
Slice 3
Air
Skin
Bone
Blood
Brain
Muscle
Temporal Muscle
Slice 5
Ω
Carlos Henrique
Ω1
Ω2
Ω3
41
Cluster Application: Human Head SAR Simulation
• SAR calculus considering energy and mass;
σ
SAR =
Ei
2ρ
2
=
Ji
2
2 ρσ
[ mW / g ]
• Applying the Poynting vector
SAR =
∂ ∂W ∂ ∂W
=
[ mW / g ]
∂t ∂m ∂t ∂V
• Local average of the SAR in terms of the simulation time (Nmax)
σ 1
SAR =
2 ρ N max
How to Build a Cluster
Carlos Henrique
∑ [E
2
x
]
+ E y2 [ mW / g ]
42
Cluster Application: Human Head SAR Simulation Performance
Tempo Despendido pela Simulação
300
250
(segundos)
Tempo Despendido
Time
(seconds)
Time of the Human Head SAR Simulation
SAR na Cabeça Humana
(dimensions variations)
200x200
200
150
100
400x400
600x600
800x800
1000x100
50
0
1
2
3
4
5
Quantidade de Processos
number
of process
Efficiency
Percentual de Eficiência
Eficiência
SAR
na Cabeça
HumanaSimulation
EfficiencydaofSimulação
the Human
Head
SAR Parallel
1,2
1
1 Processo
0,8
2 Processos
0,6
3 Processos
0,4
4 Processos
0,2
5 Processos
0
200x200
400x400
600x600
800x800
1000x1000
Dimensões dos
Domínios
Computacionais
Dimensions
of the
computational
domain
How to Build a Cluster
Carlos Henrique
43
Cluster Application: SAR Simulation varying the time steps
Eficiência
efficiency
Eficiência
SAR Simulation
Efficiencyda
in Simulação
the Human Head
SAR na Cabeça
Humana (400x400)
(400x400)
1,2
1
0,8
0,6
0,4
0,2
0
1 Processo
1 process
2 Processos
2 processes
3 Processos
3 processes
4 Processos
4 processes
5 Processos
5 processes
1000
2000
3000
4000
5000
Quantidade
de Passos
Tempo
Number
of timede
steps
How to Build a Cluster
Carlos Henrique
44
Cluster Application: SAR Simulation
Slice 1
Slice 2
Slice 5
Slice 4
How to Build a Cluster
Slice 3
Carlos Henrique
45
Cluster Application: Metamaterial simulations
Tempo Despendido
Time(seconds)
(segundos)
Tempo das Simulações
Metamaterial
2D
Metamaterial
Simulation
Time
Time (seconds)
2000
1500
1000
3000x323
AEL 1
1552
AEL2
814
AEL3
596
AEL10
468
AEL8
457
500
0
1
2
3
4
5
number
ofdeprocess
Quantidade
Processos
Eficiência do Processamento Paralelo
Metamaterial Simulation Time
1,2
efficiency
Eficiência
1
0,8
3000x323
0,6
0,4
Efficiency =
0,2
0
1
2
3
4
5
Tsequential
Tparallel
Quantidade de Processos
number
of process
How to Build a Cluster
Carlos Henrique
46
CePOF Cluster
0
1
Public Key - DSA
Internet
Router
How to Build a Cluster
Gateway
Firewall
Carlos Henrique
Intranet
47
CePOF Cluster: New Cluster Paradigm
•
15 nodes:
•
2 AMD Opteron 246 processors
•
4 GB of memory RAM
•
73 GB of HD SCSI
•
Switch 3COM 2824 (Gigabit Network)
•
Patch Cord Kat. 6
•
No break APC SURT 10000, GREE Air-Conditioner with 42.000 Btus
S.O. Virtual S.O. Virtual
Hw. Virtual Hw. Virtual
Virtual Machine
Windows 2003 Server
A2210 (AMD Opteron)
How to Build a Cluster
Carlos Henrique
48
Conclusions
• Computer Cluster allows high performance computing (HPC) with low costs;
• Free software and microcomputer reused (Beowulf cluster);
• Complex structure to configure, to use and to keep;
• Choose better Operational System distribution in order to improve the
computational performance;
• Making possible to simulate and analyze new electromagnetic structures and
many other applications.
How to Build a Cluster
Carlos Henrique
49
References
•
A. S. Tanenbaum, “Redes de Computadores 3ª. Edição”, Editora Campus, 1996.
•
C. A. F. Oliveira, “Cluster Beowulf- uma solução de baixo custo”, Revista CienteFico, Ano IV, V. I, Salvador, 2004.
•
C. H. S. Santos, H. E. Hernández-Figueroa, “Método FDTD 2D Paralelo para Simulações de Estruturas Metamateriais”,
12º Simpósio Brasileiro de Microondas e Optoeletrônica (SBMO) e 7º Congresso Brasileiro de Eletromagnetismo
(CBMAG), Belo Horizonte, 2006.
•
C. H. S. Santos, L. A. Ambrosio, H. E. Hernández Figueroa, “The Benefits of Heterogeneous Beowulf Cluster on the Human
Head SAR Simulation”, International Microwave & Optoelectronics Conference (IMOC), IEEE-SBMO, Brasília (2005)
•
C. H. S. Santos, “Computação Paralela Aplicada a Problemas Eletromagnéticos Utilizando o Método FDTD”, dissertação
de mestrado, Faculdade de Engenharia Elétrica e de Computação (FEEC), Universidade Estadual de Campinas
(Unicamp), Campinas, 2005.
•
J. J. Dongarra, et al, “Numerical Linear Algebra for High-Performance Computers”, Society for Industrial and Applied
Mathematics (SIAM), Philadelphia, 1998.
•
M. A. F. Batista, L. N. F. Guimarães, “Algoritmos Genéticos em Ambientes Paralelos”, V Worcap INPE, São José dos Campos,
2005.
•
M. Pitanga. Construindo Supercomputadores em Linux Segunda Edição, Editora Brasport, São Paulo, 2004.
•
N. S. Simões, S. F. Souza, L. Muniz, F. Fardim Junior, A. F. Souza, Reis N. C. Junior, A. M. P. Valli, L. Catabriga,
“Instalação e Configuração de Clusters de Estações de Trabalho: Experiência do Laboratório de Computação de Alto
Desempenho do Departamento de Informática da UFES”, IV Workshop em Sistemas Computacionais de Alto Desempenho WSCAD'2003, São Paulo, 2003.
•
P. S. Pacheco, “Parallel Programming with MPI”, Morgan Kaufmann Publishers, San Francisco, 1997.
•
P. Veríssimo, L. Rodrigues, “Distributed Systems for System Architects”, Kluwer Academic Publishers, Massachusetts, 2001.
•
“Introdução
ao
MPI”,
Centro
Nacional
de
Processamento
de
http://www.cenapad.unicamp.br/servicos/treinamentos/mpi.shtml (acessado em: 18/04/2007).
•
S. Humphries, “Field Solutions on Computer”, CRC Press LLC, Florida, 2000.
•
“Curso de MPI”, Centro Nacional de Processamento de Alto Desempenho-NE.
•
http://www.Beowulf.org/overview/history.html
(acessado em: 18/04/2007).
•
http://www.Beowulf.org/overview/faq.html#17
(acessado em: 18/04/2007).
•
H. Zima, B. Chapman, Supercompilers for Parallel and Vector Computers, ACM Press, New York, 1991.
How to Build a Cluster
Carlos Henrique
Alto
Desempenho-SP,
50
Thank you!
Carlos Henrique da Silva Santos
[email protected]
Leonardo André Ambrosio
[email protected]
Hugo Enrique Hernández Figueroa
[email protected]
How to Build a Cluster
Carlos Henrique
51
Network: Data Communication
Client machine
Client
process
(1) Connect request
Server machine
(2) ACK
(3) Request data
(4) Reply
System
calls
Operating
system
How to Build a Cluster
Protocol
Kernel
Drivers
stack
Carlos Henrique
Server
process
(5) Disconnect
(6) Disconnect
Kernel
Protocol
Drivers
stack
52