Good afternoon, was wondering if anyone has any tips on how to accomplish this? I have a job that I want to run a curl command on depending if a file called test.txt exists, like so:
test_job:
stage: testing
tags: Win-runner
script: |
if [ -f "test.txt" ]; then
'curl https://www.website1.com';
'curl https://www.website2.com';
My builder fails with the following error:
$ if [ -f "test.txt" ]; then # collapsed multi-line command
(https://gitlab.com/myProject/-/jobs/12304#L26) bash:
line 76: curl https://www.website1.com : command not found
Any clues as I’ve been searching the docs and the web for a solution and nothing seems to stand out.
Thanks again.
The multi-line script approach looks good. I think the problem is with the curl command execution itself.
To rule out a dependency installation problem, I’d suggest to explicitly install curl
before running any commands. I’m not sure which image or environment the job is running in, for Ubuntu/Debian based images it can look like this:
before_script:
- apt update && apt -y install curl
Although, this isn’t the problem (I spun up a test playground project). The shell interpreter treats single quoted strings as “one execution command” - to make the error more visible, we can rewrite the commands with more arguments - it will still fail. Removing all arguments will work (curl found, but missing arguments error).
test_job:
stage: test
before_script:
- apt update && apt -y install curl
script: |
if [ -f "test.txt" ]; then
'curl https://www.website1.com arg2 arg3 arg4';
'curl https://www.website2.com arg2 arg3 arg4 arg5';
Idea: Remove single quotes?
The shell interpreter should parse the command by itself, this will work. The argX passed will throw errors but they are different.
test_job:
stage: test
before_script:
- apt update && apt -y install curl
script: |
if [ -f "test.txt" ]; then
curl https://www.website1.com arg2 arg3 arg4;
curl https://www.website2.com arg2 arg3 arg4 arg5;
Without single quotes, and solution
test_job:
stage: test
script: |
if [ -f "test.txt" ]; then
curl https://www.website1.com;
curl https://www.website2.com;
More ideas
Use double quotes for variable expansion, when necessary, i.e. “curl $HTTP_ADDRESS” if you want to populate the tested value from a global variable. It should not be necessary though.
I love challenges, so I built a matrix build version of the website tests too. Maybe that’s useful for your workflows too.
matrix_test_job:
stage: test
parallel:
matrix:
- DEVSECOPS_ADDRESS: [ https://about.gitlab.com, https://opsindev.news, https://o11y.love, https://qconlondon.com ]
script: |
if [ ! -f "test.txt" ]; then
curl $DEVSECOPS_ADDRESS;
Technical background
If you are interested in learning why the shell interpreter fails with single quotes around the full command - for context, I was an OSS developer in my past job, and had to learn command parsing in C/C++ and ran into exactly these problems myself.
The underlaying technical problem is with how Linux shell execution works - the command strings are split by whitespaces, and the first argument in the list (argv[0] if you see that somewhere) being the binary to lookup, while all other commands (argv[1-n]) serve as additional command arguments. On the inside, the command parser converts this into a list of strings. The command in the solution could also be written to something like this:
'curl' 'https://www.website2.com'
If you take away the work from the shell interpreter by “wrapping” the command into single quotes already,
'curl https://www.website2.com'
the shell interpreter thinks - hey cool, the first argument (argv[0]) is already defined, no split by whitespace necessary. Passes over the full string to execvpe()
or popen()
function calls, and this is what actually fails then, causing the terminal output error message.
Unfortunately, the command parsing is not “handed” to the final command string being executed. The function that throws the error does not know how the string was passed into the terminal from the outside. Which makes these types of errors hard to read and debug. Because really, the error message should read as
line 76: 'curl https://www.website1.com' : command not found. The command seems to be wrapped in single quotes, is this intentional? Could be a potential execution problem to troubleshoot.
As developer, I’d try patching this into error messages. For curl, with many use cases, it probably is impossible to add. Different story though
Hope this little deep dive helps
Thanks for your kind feedback. I love sharing my knowledge and experience in public, so that others can benefit and learn too.
If you want to meet in person - I’m usually around the Nuremberg area, Germany, and other locations when speaking at events. I will be at QCon London in 3 weeks, and KubeCon EU in Amsterdam mid April for example.