The German D-Grid Initiative in an International Context

Transcrição

Multikonferenz Wirtschaftsinformatik, Passau 20.-22.2. 2006
Track: Betriebliche Anwendungen des P2P- und Grid-Computing
The German D-Grid Initiative
in an International Context
Wolfgang Gentzsch, D-Grid, NC-Grid, GGF
Slides are available from
[email protected]
MKWI Wirtschaftsinformatik
Passau, 20. - 22.2.2006
Topics of the Day
My favorite Grid definition
Example: The German D-Grid Initiative
The international Gridscape
10 * Grid in Context
Grid challenges and opportunities in business
Passau, 20. - 22.2.2006
Grid is a Journey . . .
Old World
New World
Static
Dynamic
Silo
Shared
Physical
Manual
Courtesy
Mark Linesch, GGF
Virtual
Automated
Service
Application
Transitioning from Silo Oriented Architecture
to
Service Oriented Architecture
Passau, 20. - 22.2.2006
The German D-Grid Initiative *)
Services for Scientists
*) funded by the German Ministry for Education and Research
Passau, 20. - 22.2.2006
German e-Science Initiative, Key Objectives
¾ Building a Grid Infrastructure in Germany
¾ Combine the existing German grid activities for infrastructure,
middleware, and applications
¾ Integration of the middleware components developed in the
Community Grids
¾ Development of e-science services for the
research community
¾ Science Service Grid
¾ Important:
¾ Continuing sustainable production grid infrastructure after
the end of the funding period
¾ Integration of new grid communities (2. generation)
¾ Business models for grid services
Passau, 20. - 22.2.2006
Inter-Project Collaboration
Im Wissensnetz
WIN-EM
WIKINGER
...
ONTOVERSE
Textgrid
Knowledge Management
MediGrid
IN-Grid
HEP-Grid
C3-Grid
Astro-Grid
D-Grid
Generic Grid Middleware and Grid Services
VIOLA
Integration Project
Passau, 20. - 22.2.2006
eSciDoc
Overview
Passau, 20. - 22.2.2006
DGI Infrastructure Project
WP 1: D-Grid basic software components, large storage,
data interfaces, virtual organizations, management
WP 2: Develop, operate and support robust core grid infrastructure,
resource description, monitoring, accounting, and billing
WP 3: Network (transport protocols, VPN)
Security (AAI, CAs, Firewalls)
WP 4: Business platform and sustainability, project
management, communication and coordination
¾ Scalable, extensible, generic grid platform for future
¾ Longterm, sustainable grid operation, SLAs based
Passau, 20. - 22.2.2006
D-Grid Middleware
User
Application
Development
and
User Access
GAT API
GridSphere
Nutzer
Scheduling
Workflow Management
High-level
Grid
Services
Monitoring
Data management
Basic Grid
Services
Plug-In
UNICORE
LCG/gLite
Accounting
Billing
User/VO-Mngt
Globus 4.0.1
Security
Resources
in D-Grid
Distributed
Data Archive
Data/
Software
Network
Infrastructur
Distributed
Compute
Resources
Passau, 20. - 22.2.2006
WP 1 Objectives
¾ Develop and implement one D-Grid middleware
infrastructure based on common grid components and
standards to provide a technology platform for resource
providers for the e-Science community
¾ Provide packaging, documentation, installation, services,
support and training for the communities
¾ Integration of the community grids with the basic grid
infrastructure
Passau, 20. - 22.2.2006
D-Grid Roadmap
Implementation:
Developing a national grid infrastructure
Middleware and applications
Services and Provider infrastructure
Realisation:
Definition of concrete projects for
infrastructure and applications
Commitment of research and
funding organizations
Concept Phase:
Identification of the partners
Definition of objectives
CY03
CY04
CY05
Passau, 20. - 22.2.2006
CY06
Key Characteristics of D-Grid
¾ Generic Grid for the German research communities
¾ Focus is on Sciences and Scientists, not industry
¾ Strong influence of existing international grid projects:
EGEE, Deisa, CrossGrid, CoreGrid, GridLab, GridCoord,
UniGrids, NextGrid, …
¾ Application-driven, not infrastructure-driven
¾ Focus on implementation, not grid research
¾ First phase: 30 MEuro 100 research organizations
¾ Embedded in larger (100 MEuro) German e-Science Initiative
Passau, 20. - 22.2.2006
The International Scenario
The Gridscape
Passau, 20. - 22.2.2006
NASA IPG Information Power Grid
(one of the first grid projects)
Enable seamless access to ground-, air-, and spacebased distributed IT resources to enable NASA missions
Computing and data Grids are emerging as the infrastructure for 21st
century science because they are providing a common way of managing
distributed computing, data, instrument, and human resources.
-- William E. Johnston, IPG Chief Architect
Passau, 20. - 22.2.2006
UK e-Science Grid: Phase 2 Web Service Grids
Edinburgh
Glasgow
DL
Belfast
Newcastle
Manchester
Oxford
Cardiff
RL
London
Soton
Cambridge
Passau, 20. - 22.2.2006
Hinxton
UK e-Science Grid
Glasgow
Belfast
Edinburgh
DL
Newcastle
Manchester
Cambridge
Oxford
Hinxton
RAL
Cardiff
London
Southampton
Application
independent
Passau, 20. - 22.2.2006
Neutron Science TeraGrid Gateway (NSTG)
John Cobb, ORNL
Passau, 20. - 22.2.2006
The Users Desktop
The Architecture of
Gateway Services
Grid
Grid
Portal
PortalServer
Server
Proxy
ProxyCertificate
Certificate
Server
Server/ /vault
vault
TeraGrid Gateway Services
Application
Application
Deployment
Deployment
Application
ApplicationEvents
Events
User Metadata
User Metadata
Catalog
Catalog
Application
Application
Workflow
Workflow
Resource
ResourceBroker
Broker
App.
App.Resource
Resource
catalogs
catalogs
Replica
ReplicaMgmt
Mgmt
Core Grid Services
Security
Security
Notification
NotificationService
Service
Accounting
Accounting
Service
Service
Policy
Policy
Resource
Resource
Allocation
Allocation
Grid Orchestration
Grid Orchestration
Reservations
Reservations
And Scheduling
And Scheduling
Data Management
Data Management
Service
Service
Administration
Administration
& Monitoring
& Monitoring
Web Services Resource Framework – Web Services Notification
Physical Resource Layer
Passau, 20. - 22.2.2006
Courtesy Jay Boisseau
National Research Grid Infrastructure
(NAREGI) 2003-2007
•
Petascale Grid Infrastructure R&D for Future Deployment
– $45 mil (US) + $16 mil x 5 (2003-2007) = $125 mil total
– PL: Ken Miura (FujitsuÎNII)
• Sekiguchi(AIST), Matsuoka(Titech), Shimojo(Osaka-U), Aoyagi (Kyushu-U)…
– Participation by multiple (>= 3) vendors, Fujitsu, NEC, Hitachi, NTT, etc.
– NOT AN ACADEMIC PROJECT, ~100FTEs
– Follow and contribute to GGF Standardization, esp. OGSA
Focused
“Grand
Challenge”
Grid Apps
Areas
Nanotech
Grid Apps
“NanoGrid”
IMS ~10TF
(Biotech
Grid Apps)
(BioGrid
RIKEN)
National AAA Infr.
Grid Middleware
R&D
(Other
Apps)
Other
Inst.
NEC
Titech
National Research
Grid Middleware R&D
Grid R&D Infrastr.
15 TF => 100TF
SuperSINET
Passau, 20. - 22.2.2006
Osaka-U
AIST
Fujitsu
IMS
U-Kyushu
Hitachi
A Grid for the Community
Example: North Carolina
NC Statewide Grid Initiative
2004 - 2006
MCNC Enterprise Grid
2003
NC BioGrid
2001
NCREN
North Carolina Research & Education Network
Passau, 20. - 22.2.2006
Australia National Grid
Darwin
Internet2
Canarie
Geant
APAN
+3500 processors
+3PB near line
storage
•10 Gbps
• IPv6
• Multicast
Perth
IVEC
CSIRO
Canberra
ANU
Sydney
Adelaide
SAPAC
GrangeNet Backbone
Centie/GrangeNet Link
AARNet Links
Brisbane
QPSF
ac3
Melbourne
VPAC
CSIRO
Passau, 20. - 22.2.2006
APAC
National Facility
Hobart
TPAC
CSIRO
The EGEE project
• Objectives
– Large-scale, production-quality infrastructure for e-Science
• leveraging national and regional grid activities worldwide
• consistent, robust and secure
– improving and maintaining the middleware
– attracting new resources and users from industry as well as science
• EGEE
– 1st April 2004 – 31 March 2006
– 71 leading institutions in 27 countries,
federated in regional Grids
• EGEE-II
– Proposed start 1 April 2006 (for 2 years)
– Expanded consortium
• > 90 partners in 32 countries
(also non-European partners)
• Related projects, incl.
– BalticGrid
– SEE-GRID
Passau, 20. - 22.2.2006
Courtesy Dieter Kranzlmueller
EGEE Applications
>20 applications from 7 domains
– High Energy Physics
– Biomedicine
– Earth Sciences
– Computational Chemistry
– Astronomy
– Geo-Physics
– Financial Simulation
Another 8 applications from
4 domains are in evaluation stage
Courtesy Dieter Kranzlmueller
Passau, 20. - 22.2.2006
EGEE gLite Services
CERN
Grid Access
Service
JRA3
EC Funded
API
UK
IT/CZ
Access Services
Information &
Monitoring
Authorization
Auditing
Application
Monitoring
Information &
Monitoring Services
Authentication
Security Services
Metadata
Catalog
File & Replica
Catalog
Storage
Element
Data
Management
Accounting
Job
Provenance
Package
Manager
Site Proxy
Computing
Element
Workload
Management
Data Services
Job Management Services
Passau, 20. - 22.2.2006
THE DEISA SUPERCOMPUTING GRID
AIX distributed
super-cluster
G
E
A
N
Vector systems
(NEC, …)
T
Linux systems
(SGI, IBM, …)
Passau, 20. - 22.2.2006
Grids for our Community: South-eastern
Coastal Ocean Observing Program
SURA
SCOOP
The Challenge:
- More than half of the nation's tidal shores, and home to 80 million people
- coastal zone is undergoing environmental and ecological changes
- threaten the sustainability of the region's economies and marine resources
The Solution:
- develop a Grid of sensors and linked computers
- fully integrating several observing systems in the southern region
- provide data, in real-time and at high speed, for more reliable, accurate and timely information
To help guide effective coastal stewardship, plan for extreme events,
facilitate safe maritime operations, and support coastal military security.
Passau, 20. - 22.2.2006
D-Grid in Context
Passau, 20. - 22.2.2006
Context #1: Benefits
Department, Enterprise, Global Grids
•
•
•
•
•
•
•
•
•
•
Resource Utilization: increase from 20% to 80+%
Productivity: more work done in shorter time
Business Agility: flexible actions and re-actions
On Demand: get resources, when you need them
Easy Access: transparent, remote, secure
Sharing: enable collaboration over the network
Failover: migrate/restart applications automatically
Resource Virtualization: access compute services, not servers
Heterogeneity: platforms, OSs, devices, software
Virtual Organizations: build & dismantle on the fly
Passau, 20. - 22.2.2006
Context #2: Standards
The Global Grid Forum
• Community-driven set of working groups that are developing
standards and best practices for distributed computing efforts
• Three primary functions: community, standards, and operations
• Standards Areas: Infrastructure, Data, Compute, Architecture,
Applications, Management, Security, and Liaison
• Community Areas: Research Applications, Industry Applications,
Grid Operations, Technology Innovations, and Major Grid Projects
• Community Advisory Board represents the different communities
and provides input and feedback to GGF
Passau, 20. - 22.2.2006
Context #3: Industry
EGA, Enterprise Grid Alliance
• Industry-driven consortium to implement standards in industry
products and make them interoperable
• Founding members: EMC, Fujitsu Siemens Computers, HP, NEC,
Network Appliance, Oracle and Sun, plus 20+ Associate Members
• May 11, 2005: Enterprise Grid Reference Model v1.0
Passau, 20. - 22.2.2006
Context #3: Industry
EGA, Enterprise Grid Alliance
• Industry-driven consortium to implement standards in industry
products and make them interoperable
• Founding members: EMC, Fujitsu Siemens Computers, HP, NEC,
Network Appliance, Oracle and Sun, plus 20+ Associate Members
• May 11, 2005: Enterprise Grid Reference Model v1.0
Feb06: GGF & EGF signed a letter of intent to
merge. A joint team is planning the transition,
expected to be complete this summer
Passau, 20. - 22.2.2006
Context #4: OGSA
ONE Open Grid Services Architecture
OGSA
Grid Technologies
Web Services
OGSA Open Grid Service Architecture
Integrates grid technologies with Web Services (OGSA => WS-RF)
Defines the key components of the grid
OGSA enables the integration of services and resources across distributed,
heterogeneous, dynamic, virtual organizations – whether within a single enterprise
or extending to external resource-sharing and service-provider relationships.”
Passau, 20. - 22.2.2006
Context #5: Quasi-Standard Tools
Example: The Globus Toolkit
• Globus Toolkit provides four major functions for building
grids
2. discover resource, MDS
3. submit job, GRAM
4. transfer data, GridFTP
1.
secure environment, GSI
Courtesy Gridwise Technologies
Passau, 20. - 22.2.2006
Globus Toolkit 4 Summary
•
Four groups of GT 4 services,
plus common libraries
•
WS components are Web
services talking SOAP
•
Non-WS components use
other protocols
Courtesy Gridwise Technologies
Passau, 20. - 22.2.2006
. . . . and
• Seamless, secure, intuitive access to distributed resources & data
• Available as Open Source
• Features: intuitive GUI with single sign-on, X.509 certificates for
AA, workflow engine for multi-site, multi-step workflows, job
monitoring, application support, secure data transfer, resource
management, and more
• In production
Passau, 20. - 22.2.2006
Courtesy: Achim Streit, FZJ
Globus 2.4 Å
UNICORE
Client
NJS
Gateway
IDB
UNICORE
UUDB
WS-Resource based
Resource Management
Framework for
dynamic resource information and
resource negotiation
TSI
GRAM Client
MDS
GRAM Gatekeeper
GRAM Job-Manager
GridFTP Client
Uspace
Globus 2
GridFTP Server
Client
Portal
Command
Line
WS-RF
WS-RF
WS-RF
RMS
WS-RF
Gateway +Gateway
Service Registry
WS-RF
WS-RF
Workflow
Engine
File
Transfer
WS-RF
User
Management
(AAA)
Network Job
Resource
Application
WS-RF
WS-RF
WS-RF
Monitoring
Management
Support
Supervisor
Passau, 20. - 22.2.2006
Courtesy: Achim Streit, FZJ
Context #6:
Global Grid Community
Passau, 20. - 22.2.2006
#7: Projects/Initiatives Testbeds
BIRN
Condor-G
Deisa
Dame
EGA
EnterTheGrid
GGF
Globus
Globus Alliance
GridBus
GridLab
GridPortal
GRIDtoday
GriPhyN
I-WAY
Knowledge Grid
Legion
MyGrid
NMI
OGCE
OGSA
OMII
PPDG
Semantic Grid
TheGridReport
UK eScience
Unicore
...
CO Grid
Compute-against-Cancer
D-Grid
DeskGrid
DOE Science Grid
EEGE
EuroGrid
European DataGrid
FightAIDS@home
Folding@home
GRIP
NASA IPG
NC BioGrid
NC Startup Grid
NC Statewide Grid
NEESgrid
NextGrid
Nimrod
Ninf
NRC-BioGrid
OpenMolGrid
OptIPuter
Progress
SETI@home
TeraGrid
UniGrids
Virginia Grid
WestGrid
White Rose Grid
...
Passau, 20. - 22.2.2006
Companies
Altair
Avaki
Axceleon
Cassatt
Datasynapse
Egenera
Entropia
eXludus
GridFrastructure
GridIron
GridSystems
Gridwise
GridXpert
HP Utility Data Center
IBM Grid Toolbox
Kontiki
Metalogic
Noemix
Oracle 10g
Parabon
Platform
Popular Power
Powerllel/Aspeed
Proxima
Softricity
Sun N1
TurboWorx
United Devices
Univa
...
FP6 Grid Technologies Projects – Calls 2, 3, 5
EU Funding: 124 M€
Call 5 start: Summer 2006
supporting the NESSI ETP & Grid community
Challengers
trust, security
Nessi-Grid
GridCoord
Grid services,
business models
Degree
Grid@Asia
GridEcon
GridTrust
AssessGrid
ArguGrid
Provenance
platforms, user
environments
Grid4all
Gredia
SIMDAT
BeinGrid
industrial
simulations
business
experiments
NextGRID
Akogrimo
BREIN
service
architecture
mobile
services
agents &
semantics
QosCosGrid
CoreGRID
g-Eclipse
GridComp
UniGrids
HPC4U
six virtual laboratories
Specific support action
Integrated project
A-Ware
Sorma
data, knowledge,
semantics, mining
XtreemOS
Linux based
Grid
operating
system
Network of excellence
Information Society and Media Directorate-General – European Commission
Unit Grid Technologies
GGF16 – Athens, 15 February 2006
39
Edutain@
Grid
KnowArc
K-WF Grid
Chemomen
tum
InteliGrid
Datamining
OntoGrid
Grid
Specific targeted research project
7th EU Research Framework Programme, FP7, 2007- 13
EU service scheme, Training
Policies, Trust
Digital Repositories, Data
Grids
Network, Supercomputers
Continuation /
enhancement of
current infrastructure
schemes
e-Infrastructure in
FP7
New research
infrastructures
(roadmap), e.g. new
supercomputing
facilities, National
Grid Centres…
Co-ordinated funding from different sources (FP7, national,
EU-structural funds, European Investment Bank loans etc
Context #9: Enterprise Grids
SunRay Access
Browser Access
via GEP
Workstation Access
Optional Control Network (Gbit-E)
Myrinet
Myrinet
Servers,
Blades,
& VIZ
Myrinet
Linux Racks
Grid Manager
Workstations
Sun Fire Link
Data Network (Gbit-E)
Gbit-E switch
V240 / V880 NFS
V240 / V880 NFS
Gbit-E switch
V240 / V880 NFS
Gbit-E switch
V240 / V880 NFS
Gbit-E switch
Gbit-E switch
V880 QFS/NFS Server
V880 QFS/NFS Server
FC Switch
NAS/NFS
Simple NFS
HA NFS
Passau, 20. - 22.2.2006
Scalable QFS/NFS
Enterprise Grid Reference Architecture
SunRay Access
Browser Access
via GEP
Access
Workstation Access
Optional Control Network (Gbit-E)
Myrinet
Myrinet
Servers,
Blades,
& VIZ
Compute
Myrinet
Linux Racks
Grid Manager
Workstations
Sun Fire Link
Data Network (Gbit-E)
Data
Gbit-E switch
V240 / V880 NFS
V240 / V880 NFS
Gbit-E switch
V240 / V880 NFS
Gbit-E switch
V240 / V880 NFS
Gbit-E switch
Gbit-E switch
V880 QFS/NFS Server
V880 QFS/NFS Server
FC Switch
NAS/NFS
Simple NFS
HA NFS
Passau, 20. - 22.2.2006
Scalable QFS/NFS
Globus Consortium
• To advance Globus Toolkit that brings an open
standards building block for enterprise grids
• Founding members: HP, IBM, Intel, Sun, Univa
• Providing resources and direction on the technology
roadmap for the Globus Toolkit by defining
specifications and requirements
Passau, 20. - 22.2.2006
1000s of Enterprise Grids in Industry
• Life Sciences
●
Electronic Design
Startup and cost efficient
Time to Market
Custom research or limited use applications
Fastest platforms, largest Grids
Multi-day application runs (BLAST)
License Management
Exponential Combinations
Well established application suite
Limited administrative staff
Large legacy investment
Complementary techniques
Platform Ownership issues
●
Financial Services
●
Market simulations
High Performance Computing
Parallel Reservoir Simulations
Time IS Money
Geophysical Ray Tracing
Proprietary applications
Custom in-house codes
Multiple Platforms
Large scale, multi-platform execution
Multiple scenario execution
Need instant results & analysis tools
Passau, 20. - 22.2.2006
Context #10 : Grid Service Providers
Pre-GRID
Example: BT
IT asset usage 10-15 %
• Inside data center, within Firewall
• Virtual use of own IT assets
• The GRID virtualiser engine inside
Firewall:
– Opens up under-used ICT assets
– improves TCO, ROI and Apps
performance
•
WANS
Intra-enterprise GRID is self limiting
– Pool of virtualised assets is restricted
by firewall
– Does not support Inter-Enterprise usage
BT is focussing on managed Grid solution
LANS
ENTERPRISE
WANS
BUT
•
ENTERPRISE
GRID Engine
LANS
Virtualised
assets
Post-GRID
IT asset usage 70-75 %
Courtesy: Piet Bel, BT
Passau, 20. - 22.2.2006
BT’s Virtual Private Grid ( VPG )
WANS
ENTERPRISE
LANS
Virtualised
IT assets
GRID Engine
WANS
ENTERPRISE A
LANS
BT NETWORK
GRID ENGINE
Courtesy: Piet Bel, BT
Passau, 20. - 22.2.2006
Context #11: There will be a Market for Grids
Passau, 20. - 22.2.2006
Business Challenges
and Potential Grid Solutions
Passau, 20. - 22.2.2006
Some Grid Challenges
Ahead
Sustainable
Competitive
Advantage
TECHNICAL
CULTURAL
LEGAL &
REGULATORY
Passau, 20. - 22.2.2006
Challenges, Example AUDI
Josef Reicheneder, AUDI
1. IT Infrastructure for distributed Engineering Data Grids
2. Data Sharing, Data Exchange, Distributed Data and Databases
3. Security of Data and Data Access
4. Encryption of Data for Collaboration with Suppliers
5. Standardised Data Representation
6. Integration of Applications (Tools)
7. Integration of different Frameworks
8. CAD Integration and Pre-processing for distributed Data Grids
9. CAT Integration for distributed Data Grids
10. Knowledge Discovery and Data Mining
Challenges, Example Intel
Ravi Subramaniam, Intel
- Most processes are a priori and manual => drive to automation
- Consistent security model (perimeter is not sufficient)
- Consistent resource descriptions (physical and logical)
- Dynamic provisioning and lifecycle management
- Embrace heterogeneity
- Global dynamic pooling and management
- Data architectures
- Interactive environments
- Integration of network characteristics
- Policy infrastructure
Business Challenges, Potential Grid Inhibitors
•
•
•
•
•
•
•
•
•
•
•
•
•
Difficult to differentiate reality from hype
Sensitive data, sensitive applications (medical patient records)
Different organizations have different ROI
Accounting, who pays for what (sharing!)
Security policies: consistent and enforced across the grid !
Lack of standards prevent interoperability of components
Current IT culture is not predisposed to sharing resources
Not all applications are grid-ready or grid-enabled
Open source is not equal open source (read the little print)
SLAs based on open source (liability?)
“Static” licensing model don’t embrace grid
Protection of intellectual property
Legal issues (FDA, HIPAA, multi-country grids)
Passau, 20. - 22.2.2006
Business Challenges, Potential Grid Inhibitors
•
•
•
•
•
•
•
•
•
•
•
•
•
Difficult to differentiate reality from hype
Sensitive data, sensitive applications (medical patient records)
Different organizations have different ROI
Accounting, who pays for what (sharing!)
Security policies: consistent and enforced across the grid !
Lack of standards prevent interoperability of components
Current IT culture is not predisposed to sharing resources
Not all applications are grid-ready or grid-enabled
Open source is not equal open source (read the small print)
SLAs based on open source (liability?)
“Static” licensing model don’t embrace grid
Protection of intellectual property
Legal issues (FDA, HIPAA, multi-country grids)
Passau, 20. - 22.2.2006
Grid Adoption in Business

Think big, start small (be a visionary and a realist)
Grid is a good opportunity to revisit existing IT infrastructure
Look for comparable success stories and use case analyses
Identify gaps, problems, concerns, … Can grids help ?
Find out in talking/listening to your users / customers !
Build YOUR list of grid business benefits and inhibitors
Create awareness -> training -> testbed (6 wks – 6 months)
Get help from Grid Service Providers
Get buy-in from upper management
Eventually, evolve from testbed into production
Passau, 20. - 22.2.2006
The Innovation Engine
Thank You !
Slides are available
[email protected]
Passau, 20. - 22.2.2006