Improvements to the AWX development experience

[gundalow]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Let's use this GitHub Issue to brain-dump problems we've seen.
Including:

• docker-compose
• Makefile targets

AWX version

devel

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

n/a

Expected results

Easier to develop.

Actual results

Not as easy as it could be

Additional information

No response

[TheRealHaoLiu]

build for ARM so people on the new M1 mac dont have to keep rebuilding the images

[gundalow]

Fail fast

• Check our dependencies are installed before we start running
  • setuptools
  • docker-compose
  • ansible
  • git

Makefile improvements

• Document main targets, as ansible-core does
• Ensure Contributing guide is clearer about which targets to use and when
• Remove older make targets

[TheRealHaoLiu]

ability to run "e2e-test" against docker-compose development environment within vscode

[TheRealHaoLiu]

ability to use vscode debugger against docker-compose development environment

vscode dev container

[Klaas-]

#13641 (comment)
"for the django migration errors, just kill the containers and rerun make docker-compose. there is some kind of race condition the first time starting the containers (when migrations are needed)."

that threw me off when I ran the tests locally :)

[fosterseth]

@Klaas- thanks for mentioning that one! we fixed it recently here #13670

[gundalow]

make ui-devel is undocumented in the README
/var/lib/awx needs to be manually created (why) with certain permissions

[shanemcd]

make ui-devel is undocumented in the README /var/lib/awx needs to be manually created (why) with certain permissions

It is documented, but almost impossible to follow the docs.

I'll start from the README and walk our way through the key parts of the documentation, and then maybe it will become more obvious how things need to get reshaped.

Starting at https://github.com/ansible/awx/blob/devel/README.md#contributing

We get taken to: https://github.com/ansible/awx/blob/devel/CONTRIBUTING.md

After some scrolling, we end up at https://github.com/ansible/awx/blob/devel/CONTRIBUTING.md#build-and-run-the-development-environment

Which sends us over to https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md

Buried deep down on that page are the instructions for building the UI: https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md#clean-and-build-the-ui

At the end of the day, as long as make / ansible / docker-compose are installed, you really just need to run:

$ make docker-compose
$ docker exec tools_awx_1 make clean-ui ui-devel

...and then wait for initial migrations to run. After that, a randomized admin password will be printed in the terminal output.

Note how we didn't need to build any image as is described in our documentation. This is because we publish a pre-built devel image which should work most of the time.

[jjwatt]

[Makefile/build]: probably a small thing, but the setuptools-scm dependency seems like a wart to me. 1. We don't support anything other than git anywhere else and 2. it fails for me all the time. One or two of the recipes have conditionals for getting it, but a lot of them fail because it's so implicit (VERSION macro always needs it, and it's at the top level). If I understand correctly, we can get the same version strings we're after by running git or checking the dir instead of relying on setuptools-scm.

[jjwatt]

[Makefile/build]: Nearly every place where the Makefile checks for VENV_BASE is wrong. They all expand to just if [ "staticstring" ] which is always true, so $VENV_BASE/bin/activate is always called.

[MarkOStewart]

There needs to be documentation and a complete package installer for people behind very strong corporate policies.
I have to download everything prior to install and try to figure out how to force it to work.

Other app/packages I have worked with release a bundle RPM (or .dev) installer that contains all of the dependency files and will install the packages in order needed.

I realize this is a big task however there are many engineers out here that will never have command line access in prod environment for git and Pypi, etc.

This issue of no complete installer combined with hundreds of old documentation repos that contain instructions to install AWX V17 and older which are completely different make for a very long untangle of a big install mess.

[AlanCoding]

Good point by @jjwatt Similar to that, the Makefile is opinionated about PYTHON. This leads to error messages when running any target which are distracting

make: python3.9: No such file or directory

Personally, I have done fairly well with my workaround which is calling things like PYTHON=$(which python) make foobar, which will accommodate the fact that I'm running in a virtual environment running with a more recent python version. However, I feel like this is one of the stumbling blocks that newcomers will encounter, as this will be a red herring for basically any other issue they may be trying to diagnose. I understand that it is important that we default to 3.9 in one scenario - I think running targets from inside the container. I have wondered if using python3 instead would still suffice for that use case without causing other problems. Maybe we could check for a version of 3.9 or higher and write errors if not.

[TheRealHaoLiu]

ability to use vscode debugger against docker-compose kube development environment

[TheRealHaoLiu]

switching branches while running development environment break the environment and cause us to start over again

(due to db migration miss match between the branch)

[TheRealHaoLiu]

make kube-devel to simply deploy kubernetes base dev environment
currently u need to do this in the operator repo rather than directly from awx repo

• clone awx-operator repo
• build/deploy awx-operator
• instantiate awx

[TheRealHaoLiu]

[program:dispatcher]
{% if kube_dev | bool %}
command = make dispatcher
directory = /awx_devel
{% else %}
command = awx-manage run_dispatcher
directory = /var/lib/awx
{% endif %}

this is annoying and clutter up our makefile

[TheRealHaoLiu]

ansible/awx-operator#1308 need this to make CI pass for awx and awx-operator on feature branches

[AlanCoding]

#13901 - we didn't even know or understand that tests do not work the same when ran in the local environment as when ran on github.

[AlanCoding]

I just updated #13521 to add some clarity to another persistent error from our Makefile. This logic throws errors on all checks, and is used to generate the version, but logically... the version must not be necessary for checks if it's not failing the checks, just throwing up a non-fatal error. I do think it will make things a lot smoother for contributors if we can clean up our own mess with things like this.

[AlanCoding]

I want to also call out #13938, which will get people accurate check results if they make pull requests against other people's forks, which sometimes happens.

[lmo5]

[Makefile/docker-compose-container-group]: Starting Kubernetes 1.24, Secrets are not automatically generated when Service Accounts are created, I suggest we create a secret of the type "service-account-token" with the same name as the service account and use that for the rest of the conf #14596

[phire-tc]

Thank you for your efforts on AWX! I've been trying to get into AWX development for a few days and it's been a bumpy journey. I'm writing this not as a screed but in the hope that it will be useful.

Starting here: https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md

The page says "clone the latest stable tag" which is 24.6.1. Note that you must create a branch, NOT check out a detached head, as the Makefile's git version sniffing breaks if you're on a detached head.

Make a branch at tag 24.6.1, then:

$ make docker-compose-build

  ... snip ...

14.29 No match for argument: openssl-3.0.7
14.31 Error: Unable to find a match: openssl-3.0.7
------
Dockerfile.dev:22
--------------------
  21 |     # Install build dependencies
  22 | >>> RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && \
  23 | >>>     dnf config-manager --set-enabled crb && \
  24 | >>>     dnf -y install \
  25 | >>>     iputils \
  26 | >>>     gcc \
  27 | >>>     gcc-c++ \
  28 | >>>     git-core \
  29 | >>>     gettext \
  30 | >>>     glibc-langpack-en \
  31 | >>>     libffi-devel \
  32 | >>>     libtool-ltdl-devel \
  33 | >>>     make \
  34 | >>>     nodejs \
  35 | >>>     nss \
  36 | >>>     openldap-devel \
  37 | >>>     # pin to older openssl, see jira AAP-23449
  38 | >>>     openssl-3.0.7 \
  39 | >>>     patch \
  40 | >>>     postgresql \
  41 | >>>     postgresql-devel \
  42 | >>>     python3.11 \
  43 | >>>     "python3.11-devel" \
  44 | >>>     "python3.11-pip" \
  45 | >>>     "python3.11-setuptools" \
  46 | >>>     "python3.11-packaging" \
  47 | >>>     "python3.11-psycopg2" \
  48 | >>>     swig \
  49 | >>>     unzip \
  50 | >>>     xmlsec1-devel \
  51 | >>>     xmlsec1-openssl-devel
  52 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf -y update && dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled crb &&     dnf -y install     iputils     gcc     gcc-c++     git-core     gettext     glibc-langpack-en     libffi-devel     libtool-ltdl-devel     make     nodejs     nss     openldap-devel     openssl-3.0.7     patch     postgresql     postgresql-devel     python3.11     \"python3.11-devel\"     \"python3.11-pip\"     \"python3.11-setuptools\"     \"python3.11-packaging\"     \"python3.11-psycopg2\"     swig     unzip     xmlsec1-devel     xmlsec1-openssl-devel" did not complete successfully: exit code: 1
make: *** [Makefile:619: docker-compose-build] Error 1

Looks like it has something to do with JIRA AAP-23449. Is there a way that I can see that?

The page says "You can skip this target if you want to use the latest ghcr.io/ansible/awx_devel:devel image rather than build a new one." Unfortunately, that crashes. See #15614

OK, there's commit 82b8f7d which looks relevant, I'll cherry-pick that and try again:

$ make docker-compose-build                                                                                                                                                                                      
ansible-playbook -e ansible_python_interpreter=python3.11 tools/ansible/dockerfile.yml \                                                                                                                                                      
        -e dockerfile_name=Dockerfile.dev \                                                                                                                                                                                                   
        -e build_dev=True \                                                                                                                                                                                                                   
        -e receptor_image=quay.io/ansible/receptor:devel

   ... snip ...

69.35 ERROR: Cannot install -r /dev/stdin (line 146), -r /dev/stdin (line 148), -r /dev/stdin (line 150), -r /dev/stdin (line 154), -r /dev/stdin (line 156), -r /dev/stdin (line 158), -r /dev/stdin (line 162), -r /dev/stdin (line 166), -r /dev/stdin (line 172), -r /dev/stdin (line 515), -r /dev/stdin (line 87), django-ansible-base[jwt-consumer,rbac,resource-registry,rest-filters]==2024.10.31.0.dev5+ga1298c0 and django==4.2.10 because these package versions have conflicting dependencies.
69.35 
69.35 The conflict is caused by:
69.35     The user requested django==4.2.10
69.35     channels 3.0.5 depends on Django>=2.2
69.35     django-auth-ldap 4.6.0 depends on Django>=3.2
69.35     django-cors-headers 4.3.1 depends on Django>=3.2
69.35     django-crum 0.7.9 depends on django>=1.8
69.35     django-extensions 3.2.3 depends on Django>=3.2
69.35     django-guid 3.2.1 depends on django<5.0 and >=3.1.1; python_version >= "3.8"
69.35     django-oauth-toolkit 1.7.1 depends on django!=4.0.0 and >=2.2
69.35     django-polymorphic 3.1.0 depends on Django>=2.1
69.35     django-solo 2.2.0 depends on django>=3.2
69.35     djangorestframework 3.15.1 depends on django>=3.0
69.35     social-auth-app-django 5.4.0 depends on Django>=3.2
69.35     django-ansible-base[jwt-consumer,rbac,resource-registry,rest-filters] 2024.10.31.0.dev5+ga1298c0 depends on Django<4.3.0 and >=4.2.16
69.35 
69.35 To fix this you could try to:
69.35 1. loosen the range of package versions you've specified
69.35 2. remove package versions to allow pip attempt to solve the dependency conflict
69.35 
69.35 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
69.35 WARNING: You are using pip version 21.2.4; however, version 24.3.1 is available.
69.35 You should consider upgrading via the '/var/lib/awx/venv/awx/bin/python3.11 -m pip install --upgrade pip' command.
70.26 make: *** [Makefile:181: requirements_awx] Error 1
------
Dockerfile.dev:62
--------------------
  60 |         /tmp/requirements/
  61 |     
  62 | >>> RUN cd /tmp && make requirements_awx
  63 |     
  64 |     ARG VERSION
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /tmp && make requirements_awx" did not complete successfully: exit code: 2
make: *** [Makefile:619: docker-compose-build] Error 1

This is mentioned in few comments like c85fa70

Commit c85fa70 looks like it might be relevant but it won't cherry-pick so I hand-applied it and now I can build. Evidently, though, I'm not finished building because there's a "Clean and build the UI" subsection that's part of the "Run AWX" section. Both of the make targets in that note (clean/ui and ui) are wrong, but I haven't figured out how that works yet so I can't suggest fixes.

A few suggestions:

• copy JIRA tickets to github and update comments so they reference tickets that everyone can see. Hidden bug trackers create a caste system
• ensure that the current devel container doesn't crash (maybe CI?)
• ensure that building the most recent release tag creates a working container, both at release-time and ongoing
• figure out why the 24.6.1 release used to build successfully but no longer does. Maybe some resources need to be pinned?

Again, this isn't a diss - AWX is a big project with a lot of moving parts so I hope that my "naive-developer" experience is helpful.