forked from cloudfoundry/docs-bosh
-
Notifications
You must be signed in to change notification settings - Fork 0
/
agent-cpi-interactions.html.md.erb
240 lines (189 loc) · 8.86 KB
/
agent-cpi-interactions.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
---
title: Agent-CPI Interactions
---
Here is an overview of the interactions between the CPI and the Agent on CloudStack as an example:
* The CPI drives the IaaS and the Agent.
* The Agent is a versatile process which configures the OS to leverage IaaS-provisioned resources (network interfaces, disks, etc.), and perform other BOSH tasks (job compilation, job instantiation, etc.)
* The CPI asks the IaaS to instantiate VM template, VMs, volumes and possibly other constructs (floating IPs, security groups, connect LBs, etc.)
* The Agent is initially driven by the CPI through the bosh-registry, and then by the Director through NATS-based messaging. The registry provides Director-side metadata to the Agent.
<div class="gray"><%= image_tag("images/cpi-interactions-overview.png") %></div>
<%#
Image source in Saas (no need to own an OmniGraffle license and run MacOS)
https://www.gliffy.com/go/html5/9205487?toke=&app=1b5094b0-6042-11e2-bcfd-0800200c9a66&dev=false
%>
The following sequence diagram illustrates in more detail the CPI, agent, and registry interactions,
in the case of the [CloudStack CPI](https://github.com/cloudfoundry-community/bosh-cloudstack-cpi-release):
<div class="gray"><%= image_tag("images/cpi-sequence-diag.png") %></div>
<%#
Image source in plantuml format.
syntax at http://plantuml.com/sequence.html
Render it online http://plantuml.com/plantuml/ or from a private plantuml instance, see http://plantuml.com/running.html
@startuml
box "DIRECTOR"
participant director
end box
box "CPI" #LightBlue
participant cpi
participant bosh_registry
end box
box "CLOUDSTACK"
participant cloudstack_api
participant vrouter
end box
box "VM" #LightBlue
participant vm
participant bosh_agent
end box
director -> cpi : create_vm;
cpi -> cloudstack_api : create vm \nand set user-data;
cpi -> bosh_registry : feed bosh registry: \n(networking, root+eph disks, \nbosh nats, blobstore
cloudstack_api -> vrouter : set user data\nin metadata\n service;
cloudstack_api -> vm : provision vm;
activate vm
vm -> bosh_agent : vm boostraps in dhcp,\nbosh-agent starts \n (reading agent.json);
bosh_agent -> vrouter : query metadata server:\nget bosh_registry address from user data;
bosh_agent -> bosh_registry : gets bootstrap info, ip adress and disks;
bosh_agent -> vm : reconfigure network static ip;
bosh_agent -> vm : mount and partition ephemeral disk;
hide footbox
@enduml
%>
---
## <a id="agent-config"></a> Agent config file formats
This section details the configuration and protocols supported by the Agent.
[VM Configuration Locations](vm-config.html#agent) provides a list of Agent configuration files and their roles.
----
### <a id="agent-json"></a> `agent.json` file
`/var/vcap/bosh/agent.json`: Start up settings for the Agent that describe how to find bootstrap settings, and disable certain Agent functionality.
The loading agent.json file is described into [config_test.go](https://github.com/cloudfoundry/bosh-agent/blob/master/app/config_test.go)
The Platform part of the file is documented into [LinuxOptions](https://godoc.org/github.com/cloudfoundry/bosh-agent/platform#LinuxOptions)
The Infrastructure part of the file is documented into [Options](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#Options) and in particular:
* the sources of configuration with [SettingsOptions](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#SettingsOptions):
* [CDROM](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#CDROMSourceOptions)
* [ConfigDrive](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#ConfigDriveSourceOptions)
* [HTTP](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#HTTPSourceOptions)
Sample `agent.json` which configures the agent to read from an HTTP metadata service at a custom URL:
```json
{
"Platform": {
"Linux": {
"CreatePartitionIfNoEphemeralDisk": true,
"DevicePathResolutionType": "virtio"
}
},
"Infrastructure": {
"NetworkingType": "static",
"Settings": {
"Sources": [
{
"Type": "HTTP",
"URI": "http://10.234.228.142"
}
],
"UseServerName": true,
"UseRegistry": true
}
}
}
```
Sample `agent.json` which configures the agent ot read from a config drive. See [reference sample config-drive json](https://github.com/cloudfoundry/bosh-agent/blob/cecf354d90f6cedf540799fb50395adb034b16d2/integration/assets/config-drive-agent.json). See [config drive integration test](https://github.com/cloudfoundry/bosh-agent/blob/cecf354d90f6cedf540799fb50395adb034b16d2/integration/config_drive_test.go):
```json
{
"Platform": {
"Linux": {
"UseDefaultTmpDir": true,
"UsePreformattedPersistentDisk": true,
"BindMountPersistentDisk": false,
"DevicePathResolutionType": "virtio"
}
},
"Infrastructure": {
"Settings": {
"Sources": [
{
"Type": "ConfigDrive",
"DiskPaths": [
"/dev/disk/by-label/CONFIG-2",
"/dev/disk/by-label/config-2"
],
"MetaDataPath": "ec2/latest/meta-data.json",
"UserDataPath": "ec2/latest/user-data.json"
}
],
"UseServerName": true,
"UseRegistry": true
}
}
}
```
----
### <a id="metadata"></a> Metadata server
The metadata server initially serves the registry URL and DNS server list.
Following is a sample content of the user-data part of the HTTP metadata
```json
{
"server": {"name": "vm-384sd4-r7re9e"},
"registry": {"endpoint": "http://192.168.0.255:8080/client/api"},
"dns": {"nameserver": ["10.234.50.180"]}
}
```
The supported format of the metadata server by the bosh-agent is documented in [UserDataContentsType](https://godoc.org/github.com/cloudfoundry/bosh-agent/infrastructure#UserDataContentsType) and [http\_metadata\_service_test.go](https://github.com/cloudfoundry/bosh-agent/blob/1dca3244702c18bf2c36483c529d4e7b3fb92b2e/infrastructure/http_metadata_service_test.go), along with the expected behavior of the bosh agent when reading this config.
----
### <a id="registry"></a> Registry
The registry provides bosh-side metadata to the bosh agent.
From the [Warden CPI documentation](https://github.com/cppforlife/bosh-warden-cpi-release/blob/be1869737c0bfba96662dde3499c9181863f91a7/docs/bosh-micro-usage.md):
* The registry is used by the CPI to pass data to the Agent. The registry is started on a server specified by registry properties.
* If SSH tunnel options are provided, a reverse ssh tunnel is created from the MicroBOSH VM to the registry, making the registry available to the agent on remote machine.
#### <a id="registry-protocol"></a> Registry HTTP protocol
The Agent expects to communicate with the bosh registry over a REST API documented in [api\_controller\_spec.rb](https://github.com/cloudfoundry/bosh/blob/2f73281f1a2a155ee807e7c0c9b8187c5a742f78/bosh-registry/spec/unit/bosh/registry/api_controller_spec.rb)
Reference registry client and servers implementations are available in:
* go-lang through [frodenas/bosh-registry](https://github.com/frodenas/bosh-registry)
* java through [cloudfoundry-community/bosh-cloudstack-cpi-core](https://github.com/cloudfoundry-community/bosh-cloudstack-cpi-core/tree/44d14a5f184d2d5e8f1f2fcd6344e734d3344673/src/main/java/com/orange/oss/cloudfoundry/cscpi/boshregistry)
#### <a id="registry-format"></a> Registry settings format
The JSON payload of the settings stored in the registry, and its format supported by the Agent are documented in [settings_test.go](https://github.com/cloudfoundry/bosh-agent/blob/ebf9a36dfddf783ffae5ae3748787d259a29374a/settings/settings_test.go)
Sample registry content:
```json
{
"agent_id": "agent-xxxxxx",
"blobstore": {
"provider": "local",
"options": {
"endpoint": "http://xx.xx.xx.xx:25250",
"password": "password",
"blobstore_path": "/var/vcap/micro_bosh/data/cache",
"user": "agent"
}
},
"disks": {
"system": "/dev/xvda",
"ephemeral": "/dev/sdb",
"persistent": {}
},
"env": {},
"networks": {
"default": {
"type": "manual",
"ip": "10.234.228.158",
"netmask": "255.255.255.192",
"cloud_properties": {"name": "3112 - preprod - back"},
"dns": [
"10.234.50.180",
"10.234.71.124"
],
"gateway": "10.234.228.129",
"mac": null
}
},
"ntp": [],
"mbus": "nats://nats:[email protected]:4222",
"vm": {"name": "vm-yyyy"},
"trusted_certs": null
}
```
----
### <a id="settings_json"></a> settings.json file
`/var/vcap/bosh/settings.json`: Local copy of the bootstrap settings used by the Agent to configure network and system properties for the VM. They are refreshed every time Agent is restarted.
The `settings.json` payload format is the same as the settings format initially returned by the registry.
---
Next: [Building a Stemcell](build-stemcell.html)
Previous: [CPI API v1](cpi-api-v1.html)