Federating your testbed with Fed4FIRE+ through the AM API

This document explains how to federate your testbed with Fed4FIRE+ through the AM API. For this we will use jFed.

First of all, note that this document is about ading an AM, an Aggregate Manager (sometimes also know as CM, “Component Manager”). If you want to setup and use your own user and slice authority (SA and/or MA), there are some extra complications not mentioned here. (Feel free to ask us for additional info if you are in this case.) Note that if you federate your AM with the Fed4FIRE+ SA and MA, there is not need to configure jFed to use the SA/MA of your testbed (if you have one), so there is also no need to configure it in jFed.

In short, this section assumes that you have a Fed4FIRE+ account, and you will login to jFed with this account, in order to use your AM.

Requirements

You need a few things before you get started:

  • A server to run the AM software on.

  • A publicly reachable IP for that server. This needs to be either an IPv4 or an IPv6 address. We recommend both.

  • A DNS name for that server, that resolves to the publicly reachable IP addresses of the server. (Recommendation: It’s nice if the DNS name refers to your testbed and is specific for your AM. Example: am.mytestbed.example.com)

    Note

    A DNS name is not strictly required, but very highly recommended

  • Choose a URN for your AM. This is of the form: urn:publicid:IDN+DNSNAME+authority+am where your replace DNSNAME by the DNS name of your AM. (Example: urn:publicid:IDN+mytestbed.example.com+authority+am).

    If you don’t have a DNS name for your server use a globally unique name for your testbed instead. It’s not recommended to use an IP address instead, though that can work. Never use localhost or 127.0.0.1, and never use the default value of the AM software you use. (For more info, see Choosing your component manager urn)

  • Choose a port at which you server will run. There is no standard port in the specification, so a lot of different ports are used in practice (12369, 8010, …). We recommend using port 443, if that is not in use by anything else. The advantage of using the default https port is that it is reachable through most firewalls, and the protocol is in fact using https.

  • You need a X.509 Server Certificate, because the AM server uses https. This can be a self signed certificate (jFed stores a list of these to make it work safely). However, in that case, make sure you configure the fields in your self signed certificate correctly. See the next section for more details.

  • You probably have testbed resources that you want to make reachable to experimenters using SSH. There are 2 options:

    • You need public IP addresses that you can assign to these nodes when needed (IPv4 or IPv6)
    • You need to have one machine with a publically reachable IP address (IPv4 recommended) act as a gateway.

    In both cases, make sure you have these address(es) available. You can combine the gateway machine with the machine on which the AM runs, but for security reasons, this is not recommended.

An example of these choices for the imec Virtual Wall2 testbed:

  • IP: 193.191.148.179
  • DNS name: wall2.ilabt.iminds.be
  • URN: urn:publicid:IDN+wall2.ilabt.iminds.be+authority+cm So the URN including the DNS name as TLD/TLA part of the URN. “authority” as “type” of the URN, and “cm” as “name” part of the URN. (We actually recommend “am” as name instead of “cm”, but both are fine.)
  • Server certificate:
    • Not self signed in our case, but signed by a trusted root.
    • “Subject Name” contains CN=www.wall2.ilabt.iminds.be, as well as details about our address and organisation
    • “Subject Alternative Name” contains DNS:www.wall2.ilabt.iminds.be, DNS:boss.wall2.ilabt.iminds.be, DNS:authority.ilabt.iminds.be
  • Our server runs at port 12369, because emulab already uses port 443 for another webserver, with different SSL settings.
  • The full URL at which our AM is reachable is: https://www.wall2.ilabt.iminds.be:12369/protogeni/xmlrpc/am/3.0
  • Experimenters can reserve servers at our AM. All of these are then reachable with SSH using public IPv6 addresses. We also have a pool of IPv4 addresses which users can assign to servers if needed. In addition, we have a gateway (bastion.test.iminds.be, which has both an IPv4 and IPv6 address), which can be used by experimenters that do not have IPv6.

Server X.509 Certificate

An AM uses a message format specified in Geni AM specification, which is sent over XML-RPC, which is sent over HTTPS with client authentication. So an AM runs at a HTTPS server (= HTTP over SSL) that is configured to require client authentication.

This means that the firsts step is to correctly setup the SSL server. That requires a server X509 certificate and matching private key. This certificate is used to identify the server to the users. Because only the server has the private key matching the certificate, users can setup a secure connection.

For SSL on the public internet, it is required that your server’s X509 certificate is signed by a trusted root. All browsers have a list of these trusted roots. This is in short how the internet is made secure.

jFed also uses the internet’s trusted roots, so if your server certificate is signed by these, that should be enough. However, for convenience, jFed also allows you to use a self signed X509 certificate. You can then add this self signed certificate to the list of certificates that jFed trusts, and everything will work without requiring the hassle of aquiring a “real” X509 certificate signed by a trusted root.

However, in recent years, thanks to “Let’s encrypt” and the excellent tool certbot it is now usually easier to get a real valid X509 certificate than it is to generate a decent self signed certificate. Because of this, we recommend setting up certbot on your server, and letting it generate the X509 certificate. Make sure that you setup certbot to automatically renew the certificate (using cron or a systemctl timer), and to restart the AM server when it does (this can be done using the certbot “post-renew hook”).

As mentioned, you can also generate a self signed certificate and register it to the central list of certificates that jFed trusts. When generating a self signed certicicate there are some best practices to take into account, regarding the self signed X509 certificate that you generate for your AM:

  • “Subject” field of the certificate must contain a “CN” that is the hostname of the server (NOT and IP, the DNS hostname!)
  • The “X509v3 Subject Alternative Name” section, must contain a “DNS” entry, which is the hostname of your server (NOT and IP, the DNS hostname!). Note that this means that your AM needs a DNS name, not just an IP address!
  • If the self-signed certificate is signed by another self-signed certificate, you need to include that other certificate in the “certificate chain” when configuring the AM’s SSL.

Note that you can also add your AM to jFed if the first 2 guidelines are not respected. However, try to at least have a meaningful “Subject” “CN”, and avoid a “Subject” “CN” like “localhost” at all cost. Also not that this will probably cause problems with non-jFed tools.

Allowing access to federated users

Federate with Fed4FIRE+ User and Slice Authority

Your AM needs to be configured so it is federated with the Fed4FIRE+ user and slice authority. How to do this depends on the AM software.

General instructions: You need to add the Fed4FIRE+ root certificate to the list of trusted authorities.

AM specific instructions:

  • For GCF, you need to put the root certificate file in the gcf trusted_roots directory (this is configurable with rootcadir in gcf_config). Example command to do this curl http://users.atlantis.ugent.be/bvermeul/wall2.pem > ~/.gcf/trusted_roots/wall2.pem

The result of configuring this is:

  • Your AM will allow (client-authenticated) SSL connections from Fed4FIRE+ users. If not configured correctly, your AM might terminate SSL handshake from Fed4FIRE+ users (so no HTTP data is even sent). If correctly configured, your AM will know for each connection that an Fed4FIRE+ user is connected. This is called “authentication”.
  • Your AM will accept user and slice “credentials” that is received from Fed4FIRE+ users. It will know what Fed4FIRE+ allows these users to do (typically, a slice credential will tell the AM a user can reserve resources). This is called “authorization”.

Firewall

You’ll need to make sure some ports are reachable:

  • Your AM is listening on one or two ports, and these should be publicly reachable.
  • If your testbed nodes are reachable directly, the necessary ssh port on each node should be publicly reachable. If you use a testbed specific ssh proxy/gateway, only that proxy should be publically reachable.
  • Your AM should be ping-able from the Fed4FIRE+ monitoring domain, so it should allow ICMP for at least that domain.
  • Your testbed OML monitoring should be able to create outgoing connections to the Fed4FIRE+ monitoring domain. (Allow outgoing TCP to flsmonitor.ilabt.imec.be:3003)

Note: The Fed4FIRE+ monitoring connections will come from:

  • 2001:6a8:1d80:2031:225:90ff:fe1d:1c68
  • 193.191.148.194

Only for ICMP you could choose to allow only these (but you can also allow public access). For AM and SSH access, the ports need to be publicly reachable anyway. Don’t rely too much on these not changing, you probably need to allow the whole subnet.

Configuring the AM GetVersion call

Your AM needs to be configured to return the correct info in the GetVersion reply. This is not a critical step, and it can be skipped. But it is a good practice to get this right. It also makes the jFed scanner work better.

The GetVersion call is a call that the scanner uses to retrieve basic server info. It is defined in the AM specification. The results contains a lot of info, most of which will be automatically filled in correctly by the server software. Experience shows that the following fields have to be checked for correctness:

"urn": "urn:publicid:IDN+example.com+authority+am",
"geni_api_versions": {
    "2": "https://example.com/am/2",
    "3": "https://example.com/am/3"
}

You need to make sure that:

  • The URN is correct. That includes the hostname part (example.com in the example above), and the name part am in the example above. The name should be am, or cm. The URN must always match the expected URN in the component_manager_urn attribute in the request RSpec.

  • The geni_api_versions URLs must point to the correct server URL(s). This must be the URL at which the AM is publically available. Make sure that:

    • The protocol is https
    • The port is correct
    • The path is correct
    • The hostname is not localhost or 127.0.0.1.
    • Preferably, the hostname is not an IP address but a DNS hostname.
    • Multiple URLs are allowed if the server supports multiple AM versions. But this is not required. Do make sure however that at least the version contacted is listed. That is, if you connect to https://example.com/am/2 it is at least expected that "2": "https://example.com/am/2" is listed.

Using the jfed scanner tool

Note

This section assumes you are using linux or MAC. Instructions for windows are very similar.

The jFed scanner can be used to automatically detect basic AM server settings, and store them for use by the other jFed tools. This way, you can add use an experiment AM to your local jFed, and test it.

While the scanner can be a usefull tool, it fails in some scenarios (as it is hard to test all strange AM configurations). It is also not frequently tested, so it can break in some releases. Finally, it is not aimed at end users, and is thus not a user friendly tool. For these reasons, if you experience problems with it, don’t hesitate to contact us.

You can find the current stable jFed Scanner GUI in this archive.

After downloading the archive, and extract it to a directory. Then execute scanner-gui.jar.

Example commands:

 $ wget -nv https://jfed.ilabt.imec.be/releases/develop/VERSION/jar/jfed_gui.tar.gz
2017-01-10 11:27:28 URL:https://jfed.ilabt.imec.be/releases/develop/VERSION/jar/jfed_gui.tar.gz [25654644/25654644] -> "jfed_gui.tar.gz" [1]
 $ tar xfz jfed_gui.tar.gz
 $ cd jfed_gui
 $ java -jar scanner-gui.jar

NOTE: Replace VERSION in the command above with a jFed version. You can find a link to the stable version at https://jfed.ilabt.imec.be/downloads/ or you can pick a development version at https://jfed.ilabt.imec.be/releases/develop/?C=N;O=D

Once the scanner tool has started, log in using your Fed4FIRE+ account.

In the next screen, fill in the Aggregate Manager URL, and click the Scan button.

Requirements for the URL:

  • It needs to be HTTPS
  • Do not forget the correct port
  • Do not forget to add the full path to the AM endpoint
  • Preferably, use a DNS hostname, not an IP address

When the scan is completed, you end up on the Scan Overview tab. Here, in the ideal case, you’ll see 4 green OK statuses. If not, something went wrong. In the Call Logs and Debug Logs tab, you can find debug info which can help when something goes wrong.

In the Scan Output tab, all information the scanner gathered is shown. It is sometimes needed to uncheck the checkboxes next to any incorrect or uneeded information. But usually, you do not need to do anything here.

In the Security tab, you will find the X509 certificate of the server. You need to verify if this is indeed the certificate of your server, and if that is so, you need to check the checkbox next to I trust all of the certificates above.

Then, click on the Show Results button.

You will see a JSON snippet containing the info that jFed will use. You can edit the following fields:

  • id: This should be a very short unique ID for the testbed. It should only contain letters, no spaces or special characters.
  • longName: This is the “human readable” name of your testbed. This name may contain spaces and special characters.
  • server/name: This is the “human readable” of the testbed server. This name may contain spaces and special characters. (It can be the same as the testbed longName)
  • defaultComponentManagerUrn: This is the URN of your AM. Make sure it is correct.

If you click on Add to local jFed, the JSON snippet will be stored in ~/.jFed/extra_testbeds/<id>.json (where <id> is the ID you chose for the testbed). All jFed tools will load files from that dir to add additional testbeds.

Local files for adding testbeds

Using local files, you can add your AM to jFed.

When the jFed scanner has succesfully scanned a server, it shows a button “Add to local jFed”, which will write such a file.

These files are stored in the jFed config dir, in the extra_testbeds dir. On Linux this is in ~/.jFed/extra_testbeds/.

Below is an example of such a file with dummy values. You can also look at the global jFed configuration (see link in next section) for real examples.

Content of ~/.jFed/extra_testbeds/mytestbed.json:

{
    "id": "mytestbed",
    "longName": "My Brand New Testbed",
    "defaultComponentManagerUrn": "urn:publicid:IDN+mytestbed.org+authority+am",
    "allowLinks": false,
    "servers": [
        {
            "name": "My Brand New Testbed",
            "certificateChain": "-----BEGIN CERTIFICATE-----\nMII... SERVER CERTIFICATE HERE ...\n-----END CERTIFICATE-----\n",
            "urnTld": "mytestbed.org",
            "baseUrl": "https://am.mytestbed.org/",
            "flags": [
                "featureExecuteAndInstallService",
            ],
            "services": [
                {
                    "api": "Geni.AM",
                    "apiVersion": "3",
                    "url": "https://am.mytestbed.org/",
                    "urn": "urn:publicid:IDN+mytestbed.org+authority+am",
                    "@type": "Service"
                }
            ],
            "defaultComponentManagerUrn": "urn:publicid:IDN+mytestbed.org+authority+am",
            "@type": "Server"
        }
    ],
    "@type": "Testbed"
}

Adding to jFed for all users

The jFed team can also add the same info about your AM, to the “global” jFed config. That enables the use of your AM for all jFed users. The global jFed configuration is available at https://flsmonitor-api.fed4fire.eu/testbed?embed=true

Contact us to add your AM.

You do not need to have a local wokring config, but that helps.

The minimal info you need to send us is:

  • The URL at which your AM runs (the full URL, including hostname, port and path)
  • The name you want to see for your testbed inside jFed and at our monitoring site
  • An example request RSpec (which shows us the sliver_type(s) used and other info)
  • Does your testbed support (or even require) specifying disk images in the request RSpec?
  • Does your testbed support (or even require) assigning specific nodes (component_id attibute) in the request RSpec?
  • Does your testbed support (or even require) specifying hardware types in the request RSpec?
  • Are links between nodes of your testbed supported?
  • Does your testbed support “stitched” links? (If you’re unsure what that is, your testbed doesn’t support them.)

Other info which is usefull to send us is:

  • Which “icon” in the jFed experiment GUI is the best match for your testbed nodes (ex: physical node, VM, wireless, …)?
  • For security reasons, it is recommended to also send your server certificate (in PEM format). We can also retrieve the certificate ourself, but theoretically, that could be intercepted.
  • The coordinates of the testbed (latitude, longitude and country)
  • Info on the organisation running the testbed (full name, latitude, longitude, country, address, link to logo, link to website)
  • Additional info about your testbed: link to documentation, link to testbed description and/or other general testbed information.
  • If your testbed has a web interface (where users can login and control the testbed) in addition to the AM, you can send us that link as well.
  • The email address(es) of the “primary” contact(s) of the testbed. If we need to contact someone about the testbed, we use these emails.
  • The email address(es) of the “technical” contact(s) of the testbed. For technical questions about the testbed, these emails are used. These emails also are used for automatic mails about the tests, and in the future, testbed specific bugreports will also be sent here. Let us know wether or not you want the technical contact(s) of the testbed to receive automated emails when the testbed goes down.
  • The email address(es) of the GDPR contact(s) of the testbed. Anything related to the GDPR we will send here.
  • The URN(s) of the users that may have admin access to some of the data we store about your testbed. This is currently not used, but we plan on using this later to allow you to restart tests, enable/disable tests or edit testbed info. This is typically the URN of the users you use to login to jFed.
  • Which software does your testbed AM use? Typical options are: emulab, gcf, openstack, …

(A lot of email addresses are mentioned above, but typically, they are all just the same single email address.)

Adding your AM to one of the experiment GUI “icons”

The jFed Experimenter GUI has a graphical editor, where you can drag “icons” to the canvas and in this way easily configure an experiment. Each of these icons has their own list of relevant testbeds.

Note that you do not need these icons to test your AM using the jFed Experimenter GUI: after you added your testbed to jFed, you can directly edit the XML Rspec, and everything will work.

It is currently not possible to manually add a testbed to one of these jFed icons in your local jFed install. Adding testbeds to the “icons” can only be done by the jFed developers. When you contact us to add your testbed to jFed for all users, we will discuss which icon(s) it needs to be added too, and add it for you.

Typically, to allow you to test without adding the testbed to the icon for all users, we add it to “dev-testbeds”. Icons added here are not available to all users. To show these testbeds in jFed, go to “Preferences”, then select “GUI Editor” and enter “dev-testbeds” in the “Extra Flags” field. Now restart jFed, and you will be able to select the testbed when using the appropriate icon.

jFed and Testbed specific documentation

The jFed experimenter GUI shows links to the testbed documentation. This makes it easy for experiments to find testbed documentation.

Two links can be set in the central jFed config, which will both be shown in jFed:

  • “Testbed Info” button: a link to a (short) general description of the testbed
  • “Testbed Documentation” button: a link to a site that has step by step instructions on how to use the testbed from jFed. A short and simple “getting started” tutorial is ideal. It’s off course also nice if this site contains links to more info and more advanced tutorials.

These links are stored in the central jFed config, and testbed owners can ask us to add/update it. In future jFed versions, this link might be shown more prominently.