README.md 16.1 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1
2
ioppytest framework:
--------------------
Federico Sismondi's avatar
Federico Sismondi committed
3

4
ioppytest is a framework for building interoperability testing tools.
Federico Sismondi's avatar
Federico Sismondi committed
5
6
7
8
9

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Handles requirements coming from:

- interop testing best practises for network protocols
- distributed execution of the tests (IUTs may be in remote locations)
- test coordination between users (user-assisted test)
- driving automated IUTs, in a generic, program-language-agnostic way
- tunneling mechanism for enabling remote interop test over
distant locations and bypass middle-box.

Some of the features include:
tunneling, sniffing, test coordination, dissection, traces analysis,
test spec online server, etc. Each of these are independent python
modules, they communicate using json messages over AMQP.

For more about this check out the AMQP API defined in the `messages`
package which is installed with
(ioppytest-utils)*[https://pypi.org/project/ioppytest-utils/]:


Test suites
-----------

currently ioppytest includes interop tests for:

- coap
- 6lowpan
- onem2m
- lwm2m
- wot
Federico Sismondi's avatar
Federico Sismondi committed
39

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
see http://doc.f-interop.eu/testsuites/

ioppytest is parametrized for running the different test suites using
the files in env/<test_suite_folder> and ioppytest/test_descriptions

Either if you are a testing tool developers or test suite users,
you can opt to disable some features. You can do this just by modifying
supervisor.conf.ini in env/<test_suite_folder> directory.

How can I use it?
-----------------

You can either go to (F-Interop Platform)*[https://go.f-interop.eu]
which builds and deploys the tool automatically for you, this provides
also a nice looking web-based GUI (recommended).

Run in a stanalone less user friendly way. Some docker images builds,
and setting up of a RMQ server is needed. User will then use the CLI
for interfacing with the testin tool.

for more info about the standalone deployment, continue reading..
Federico Sismondi's avatar
Federico Sismondi committed
61

Federico Sismondi's avatar
Federico Sismondi committed
62
63
Implemented test suites in the ioppytest framework:
---------------------------------------------------
Federico Sismondi's avatar
Federico Sismondi committed
64

Federico Sismondi's avatar
Federico Sismondi committed
65
66
The test suites implemented which are currently supported by the
framework are:
Federico Sismondi's avatar
Federico Sismondi committed
67

Federico Sismondi's avatar
Federico Sismondi committed
68
69
70
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
71
72
- LwM2M Test Suite (between two users' IUT) (WIP)
- oneM2M Test Suite (between two users' IUT) (WIP)
Federico Sismondi's avatar
Federico Sismondi committed
73

Federico Sismondi's avatar
Federico Sismondi committed
74
75
Test setup:
-----------
Federico Sismondi's avatar
Federico Sismondi committed
76

Federico Sismondi's avatar
Federico Sismondi committed
77
78
79
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
80

Federico Sismondi's avatar
Federico Sismondi committed
81
82
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
83
84

```
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

                                                                        +----------------------------+
                                                                        |                            |
                                                                        |    ioppytest Test Tool     |
                                                                        |(CoAP, 6LoWPAN, OneM2M, etc)|
                                                                        |                            |
                                                                        |                            |
                                                                        +----------------------------+

                                                                                    ^    +
                                                                                    |    |
                                                                                    |    |
                                                                                    |    |
                                                            packet.fromAgent.agent_x|    |  packet.toAgent_agent_y
                                                                                    |    |
                                                            packet.fromAgent.agent_x|    |  packet.toAgent_agent_y
                                                                                    |    |
                                                            ui.user_1.reply         |    |  ui.user_1.request
                                                                                    +    v

                +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                                 AMQP Event Bus
                <-------------------------------------------------------------------------------------------------------------------------------------------------+
                      |   ^                              ^  |                                           |   ^                              ^  |
                      |   |     packet.fromAgent.agent_x |  | packet.toAgent_agent_x                    |   |     packet.fromAgent.agent_y |  | packet.toAgent_agent_y
             ui.user_1|   |                              |  |                                           |   |                              |  |
              .request|   | ui.user_1.reply              |  |                          ui.user_2.request|   | ui.user_2.reply              |  |
                      |   |                              |  |                                           |   |                              |  |
                      |   |                              |  v                                           |   |                              |  v
              +-------v-------------+      +--------------------------------+                   +-------v-------------+      +--------------------------------+
              |                     |      | +---------------------------+  |                   |                     |      | +---------------------------+  |
              |   User Interface    |      | |      Agent (agent_x)      |  |                   |   User Interface    |      | |      Agent (agent_y)      |  |
              |       (user 1)      |      | |        (tun mode)         |  |                   |       (user 2)      |      | |        (tun mode)         |  |
              |                     |      | +---------------------------+  |                   |                     |      | +---------------------------+  |
              +---------------------+      |                                |                   +---------------------+      |                                |
                                           | +-----+tun interface+-------+  |                                                | +-----+tun interface+-------+  |
                                           |                                |                                                |                                |
                                           | +----------------------------+ |                                                | +----------------------------+ |
                                           | | Implementation under test  | |                                                | | Implementation under test  | |
                                           | |          using IP          | |                                                | |          using IP          | |
                                           | |      (e.g. coapoclient)    | |                                                | |      (e.g. coaposerver)    | |
                                           | +----------------------------+ |                                                | +----------------------------+ |
                                           +--------------------------------+                                                +--------------------------------+



131
132
```

Federico Sismondi's avatar
Federico Sismondi committed
133
134
135
136
137
138
139
Event Bus API:
--------------

All the calls between the components are documented here:

[interop tests API doc](http://doc.f-interop.eu/interop/)

140

Federico Sismondi's avatar
Federico Sismondi committed
141
142
Running a test suite:
---------------------
Federico Sismondi's avatar
Federico Sismondi committed
143

Federico Sismondi's avatar
Federico Sismondi committed
144
user needs :
Federico Sismondi's avatar
Federico Sismondi committed
145

Federico Sismondi's avatar
Federico Sismondi committed
146
- an implementation under test (IUT) of a standard supported/protocol
147
by ioppytest framework e.g. a coap client implementation
148
-  run
Federico Sismondi's avatar
Federico Sismondi committed
149
[the agent component](http://doc.f-interop.eu/interop/#agent)
150
151
152
which plays the role of a vpn-client , and which will route all the
packets sent from the IUT (on a certain ipv6 network) to the
backend -which is later on routed to second IUT (and viceversa)-.
Federico Sismondi's avatar
Federico Sismondi committed
153
154
155
- a user interface to help coordinating the tests
(either GUI or CLI component)

156
For simplifying the access to CLI, agent and other components, ioppytest
157
includes a Makefile, with it you can use `make <cmd>`,
158
159
for more information execute `make help`

160
### make commands
Federico Sismondi's avatar
Federico Sismondi committed
161

162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
```
➜  ioppytest git:(master) ✗ make help
help:                          Help dialog.
build-all-coap-images:         Build all testing tool in docker images, and other docker image resources too
build-tools:                   builds all testing tool docker images (only testing tool)
build-automated-iuts:          Build all automated-iut docker images
build-all:                     Build all testing tool in docker images, and other docker image resources too
clean:                         clean data directory
sniff-bus:                     Listen and echo all messages in the event bus
run-cli:                       Run interactive shell
run-6lowpan-testing-tool:      Run 6LoWPAN testing tool in docker container
run-coap-testing-tool:         Run CoAP testing tool in docker container
run-lwm2m-testing-tool:        Run lwm2m testing tool in docker container
run-onem2m-testing-tool:       Run oneM2M testing tool in docker container
run-comi-testing-tool:         Run CoMI testing tool in docker container
stop-all:                      stops testing tools and IUTs running as docker containers
validate-test-description-syntax: validate (yaml) test description file syntax
run-tests:                     runs all unittests
get-logs:                      echoes logs from the running containers
install-python-dependencies:   installs all py2 and py3 pip dependencies
```
Federico Sismondi's avatar
Federico Sismondi committed
183

184
# Test session setups:
Federico Sismondi's avatar
Federico Sismondi committed
185

186
The supported setups are:
Federico Sismondi's avatar
Federico Sismondi committed
187

188
- user drives one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
189
automated-IUTs the framework supports
190
- user drives both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
191
session
192
193
- user1 drives an IUT, user2 drives an IUT, users are either both
in-situ, or remotely located.
Federico Sismondi's avatar
Federico Sismondi committed
194

195
# (opt 1) Running a test suite using F-Interop platform
196
197
198

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

199
200
201
Recommended option (more user friendly).

# (opt 2) Running a test suite standalone
Federico Sismondi's avatar
Federico Sismondi committed
202

203
204
205
for this, you will use ioppytest_cli  as CLI for
interfacing with testing tool (comms go over AMQP event bus).

206
Recommended option only for testing tool contributors.
207

208
## (opt 2.1) Set up up the message broker
209
210
211

The interop testing tool use RabbitMQ (RMQ) message broker for sending
messages between its components, and the remote ones (like the agent).
Federico Sismondi's avatar
Federico Sismondi committed
212
213

RMQ broker is a component which is **external** to the testing tool
214
and which establish the session and infrastructure so components can
Federico Sismondi's avatar
Federico Sismondi committed
215
communicate with each other during the test session.
Federico Sismondi's avatar
Federico Sismondi committed
216

217
218
219
220
221
222
If using [go.f-interop.eu](go.f-interop.eu) then this is automatically
set-up for you.

When running a standalone setup the user first needs to have a RMQ
broker running..

223
224
225
226
227
228
229
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
230
- Request a remote vhost and credentials (user,pass) to
231
federico<dot>sismondi<at>inria<dot>fr (recommended)
Federico Sismondi's avatar
Federico Sismondi committed
232
233
234

don't hesitate to contact me, this is a very simple procedure and it's
free :D
235

236
## (opt 2.2) Export AMQP environment variables
237

238
239
240
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
Federico Sismondi's avatar
Federico Sismondi committed
241

Federico Sismondi's avatar
Federico Sismondi committed
242
243
244
245
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
246

247
## (opt 2.3) Download the source code (see `Clonning the project` for more info)
Federico Sismondi's avatar
Federico Sismondi committed
248
```
249
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
250
251
cd ioppytest
```
252
253


254
## (opt 2.4) Build the testing tools
255

Federico Sismondi's avatar
Federico Sismondi committed
256
257
(docker, py2 and py3 needs to be installed in the machine)
```
258
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
259
```
260

261
## (opt 2.5) Run testing tool (CoAP testing tool example)
Federico Sismondi's avatar
Federico Sismondi committed
262
263
264
```
make run-coap-testing-tool
```
265

266
## (opt 2.6) Connect to the interop session using the CLI
Federico Sismondi's avatar
Federico Sismondi committed
267
268
269
```
make run-cli
```
270

271
## (opt 2.7) Connect the agent to the backend
272

Federico Sismondi's avatar
Federico Sismondi committed
273
if user's IUT is a CoAP client:
274

Federico Sismondi's avatar
Federico Sismondi committed
275
276
277
```
make run-agent-coap-client
```
278

Federico Sismondi's avatar
Federico Sismondi committed
279
if user's IUT is a CoAP server:
280

281
```
Federico Sismondi's avatar
Federico Sismondi committed
282
make run-agent-coap-server
283
284
```

285
## (opt 2.8) Running a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
286

287
### (opt 2.8.1) User to user session, second user with his/her own IUT
Federico Sismondi's avatar
Federico Sismondi committed
288
289
290
291
292
293
294
295

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

296
### (opt 2.8.2) Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
297
298
299

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
300

301
```
Federico Sismondi's avatar
Federico Sismondi committed
302
make run-coap-server
303
304
```

Federico Sismondi's avatar
Federico Sismondi committed
305
306
or for a coap client automated implementation:

307
```
Federico Sismondi's avatar
Federico Sismondi committed
308
make run-coap-client
309
310
```

Federico Sismondi's avatar
Federico Sismondi committed
311

312
## (opt 2.9) Running the interop session
Federico Sismondi's avatar
Federico Sismondi committed
313
314
315
316
317
318
319
320
321
322
323
324

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
325
```
326
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
327
cd ioppytest
328
329
```

Federico Sismondi's avatar
Federico Sismondi committed
330
331
332
333
334
335
336
337
338
339
## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
340

341
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
342

343
```
Federico Sismondi's avatar
Federico Sismondi committed
344
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
345
346
347
348
349
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
350
351
```

352
353

alternatively, if you are curious and you want to know
354
355
what's under the hood, you can see which processes are being run -in
the docker container- by the testing tool:
Federico Sismondi's avatar
Federico Sismondi committed
356

357
```
Federico Sismondi's avatar
Federico Sismondi committed
358
359
360
361
362
363
364
365
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

366
367
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
368
369
370
371
372
373
374
375
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
376
377
or you can also run directly the processes without docker:
(supervisord needed)
378

Federico Sismondi's avatar
Federico Sismondi committed
379
380
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
381
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
382
```
383

Federico Sismondi's avatar
Federico Sismondi committed
384
385
386
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
387
note: use -E when launching supervisor process, it preserves the
388
env vars (like an exported AMQP_URL)
389
390


Federico Sismondi's avatar
Federico Sismondi committed
391
392
FAQ
---
393

Federico Sismondi's avatar
Federico Sismondi committed
394
- How can I install docker on my machine?
395

Federico Sismondi's avatar
Federico Sismondi committed
396
    For this just follow this instructions: https://docs.docker.com/engine/installation/
397

Federico Sismondi's avatar
Federico Sismondi committed
398

Federico Sismondi's avatar
Federico Sismondi committed
399
400
401
402
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
Federico Sismondi's avatar
Federico Sismondi committed
403

Federico Sismondi's avatar
Federico Sismondi committed
404
405
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
Federico Sismondi's avatar
Federico Sismondi committed
406

407
    please contact federico<dot>sismondi<at>inria<dot>fr
408
409


Federico Sismondi's avatar
Federico Sismondi committed
410
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
411

Federico Sismondi's avatar
Federico Sismondi committed
412
    try using ```--no-cache``` for the docker build
413

Federico Sismondi's avatar
Federico Sismondi committed
414
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a