When dealing with environments that should meet security regulations it is very unlikely that systems have direct access to the Internet: they connect to the Internet by the means of a content proxy.
When dealing with the need of access the online Ansible Galaxy, you may think that creating the access rule on the corporate proxy is enough to sort things out, but it isn't: besides the poor performances due to the latency for downloading contents from the Internet, there's also the security concern that it is possible to upload contents to Ansible Galaxy using that proxy.
If you are working in an environment with sensitive data, this is a huge security risk that you can mitigate by using an on premise repository manager that mirrors the contents you need from the online Ansible Galaxy.
This post shows you how easily you can mirror a subset of the collections hosted onto the online Ansible Galaxy by using Pulp3 with the Ansible plugin.
First and foremost we should install Ansible: enable the EPEL repository
sudo dnf install epel-releaseand eventually install Ansible:
sudo dnf install ansiblewe are now ready to develop the playbooks necessary to manage everything.
We want to configure and manage everything using a YAML formatted manifest like the following one:
---
repositories:
- name: "galaxy.ansible.com"
description: "The official Ansible Galaxy online repository"
base_path: "galaxy.ansible.com"
remote:
name: "galaxy.ansible.com"
url: "https://galaxy.ansible.com/"
requirements_file: "collections:\n - vmware.vmware_rest\n - cloud.common"save it as repositories.yml: it is a list of dictionaries we can use to define one ore more repositories along with their remote.
In Pulp3 terms, a remote is an object that contains all the information necessary to locate and download contents from a remote repository: in this example we download only vmware.vmware_rest and cloud.common collections from https://galaxy.ansible.com remote repository.
Create a few var files where to store different kind of settings:
the first is vars/defaults.yml
---
remotes_base_path: "/pulp/api/v3/remotes"
repositories_base_path: "/pulp/api/v3/repositories"
distributions_base_path: "/pulp/api/v3/distributions"
plugins:
- "ansible"
ansible_remote_kinds:
- role
- collection
remotes_plugin_path:
ansible: "ansible"
repositories_plugin_path:
ansible: "ansible/ansible"
distributions_plugin_path:
ansible: "ansible/ansible"
here we define some variables along with the necessary defaults to implement input validation and to create a code that scale well: we are focusing on Ansible Pulp3 plugin, but as we saw Pulp3 supports other type of repository too- we don't want to have to re-work the code to add the support to the other type of repository, do we?
The second file is vars/settings.yml:
---
base_addr: http://{{ ansible_fqdn }}:8080
username: "admin"
here we define some settings the user may want to change to suit it's actual topology and Pulp3 settings.
The last file is vars/secrets.yml:
---
password: "admin"
since this file contains sensitive data, it should be encrypted by ansibile-vault.
For the sake of simplicity in this post I keep it plain-text, but in real life scenarios remember to encrypt it with ansible-vault, or use any kind of thirdly part vault.
Now create the add-repos.yml playbook:
---
- name: create a remote
hosts: localhost
become: false
vars_files:
- vars/settings.yml
- vars/defaults.yml
- vars/secrets.yml
- repositories.yml
tasks:
- name: fail if repo type is missing
fail:
msg: "You shoul define 'type' variable to specify the type of repo you want to create"
when: type is not defined
- name: fail on unsupprted repo type
fail:
msg: "the type {{ type }} is invalid or not supported yet"
when: type not in plugins
- name: fail if kind is missing
fail:
msg: "You shoul define 'kind' variable to specify the kind of remote you want to create"
when:
- type == "ansible"
- kind is not defined
- name: fail on unsupprted remote kind
fail:
msg: "the kind {{ kind }} is invalid or not supported yet"
when:
- type == "ansible"
- kind is defined
- kind not in ansible_remote_kinds
- name: create repositories
include_tasks: add-repo.yml
loop: "{{ repositories }}"
It loops over each repository declared in the YAML manifest file using the tasks defined in add-repo.yml.
So now create add-repo.yml:
---
- set_fact:
remotes_uri: "{{ base_addr }}{{ remotes_base_path }}/{{ remotes_plugin_path[type] }}/{{ kind }}/"
when: kind is defined
- set_fact:
remotes_uri: "{{ base_addr }}{{ remotes_base_path }}/{{ remotes_plugin_path[type] }}/"
when: kind is not defined
- name: check if {{ item.remote.name }} does already exists
uri:
url: "{{ remotes_uri }}"
method: GET
status_code: 200
return_content: true
user: "{{ username }}"
password: "{{ password }}"
force_basic_auth: yes
register: remotes_response
- fail:
msg: "remote '{{ item.remote.name }}' does already exist"
when: remotes_response.json.results | json_query(jmesquery) |length > 0
vars:
jmesquery: "[? name=='{{ item.remote.name }}' ].pulp_href"
- name: create remote {{ item.remote.name }}
uri:
url: "{{ remotes_uri }}"
method: POST
status_code: 201
return_content: true
user: "{{ username }}"
password: "{{ password }}"
force_basic_auth: yes
body_format: "json"
body: "{{ item.remote }}"
register: remotes_response
- name: create {{ item.name }} repository
uri:
url: "{{ base_addr }}{{ repositories_base_path }}/{{ repositories_plugin_path[type] }}/"
method: POST
status_code: 201
return_content: true
user: "{{ username }}"
password: "{{ password }}"
force_basic_auth: yes
body_format: "json"
body: '{ "name": "{{ item.name }}", "description": "{{ item.description }}", "remote":"{{ remotes_response.json.pulp_href }}" }'
register: repositories_response
- name: create {{ item.name }} distribution
uri:
url: "{{ base_addr }}{{ distributions_base_path }}/{{ distributions_plugin_path[type] }}/"
method: POST
status_code: 202
return_content: true
user: "{{ username }}"
password: "{{ password }}"
force_basic_auth: yes
body_format: "json"
body: '{ "name": "{{ item.name }}", "base_path": "{{ item.base_path }}", "repository":"{{ repositories_response.json.pulp_href }}" }'
register: distributions_response
- name: check {{ item.name }} distribution creation job state
uri:
url: "{{ base_addr }}{{ distributions_response.json.task }}"
status_code: 200
return_content: true
user: "{{ username }}"
password: "{{ password }}"
force_basic_auth: yes
register: distribution_creation_result
until: distribution_creation_result.json.state != "running"
retries: 20
delay: 3
- debug:
msg:
- "something went wrong while synchronizing the remote source. Please retry"
- "{{ distribution_creation_result.json.error.description }}"
- "{{ distribution_creation_result.json.error.traceback }}"
when: distribution_creation_result.json.error.traceback is defined
failed_when: distribution_creation_result.json.state != "completed"
- debug:
msg: "Repository, Remote and Distribution {{ item.name }} creation completed"
- lines from 2 to 7 are used to define the remote_uri variable with a different value whether the "kind" token has been or hasn't been supplied - The "kind" token is used by Ansible repository type, but other repository types does not need to specify the kind of remote.
- lines from 8 to 22 check if the specified remote does already exists, making the play to fail in this case: this is to ensure that the user do not run the create playbook multiple times by mistake
- lines from 23 to 34 create the remote, and store the output returned by the call to the API into remotes_response variable
- lines from 35 to 46 create the repository, and store the output returned by the call to the API into repositories_response variable
- lines from 47 to 79 create the distribution; this is a complex operation that is run using an asynchronous task:
- the first call creates the task (lines from 47 to 58)
- the second call keeps polling the created task until it completes and stores the API output of the check into distribution_creation_result variable (lines from 59 to 70)
- the last two tasks checks the outcome of the call (lines from 71 to 79)
run the add-repos.yml Ansible playbook - specify that we want to create an Ansible repo (-e type=ansible) and that the remote should synchronize collections (-e kind=collection):
ansible-playbook -i inventory -e type=ansible -e kind=collection add-repos.ymlat the end of the play, the outcome should be as follows:
tbd