Target upload_dirs do not have any files in them inside container, when using mutagen after a ddev pull pantheon or ddev import-files

[tesliker]

Preliminary checklist

• I am using the latest stable version of DDEV (see upgrade guide)
• I have searched existing issues
• I have checked the troubleshooting guide
• I have run ddev debug test to include output below

Output of ddev debug test

Expand `ddev debug test` diagnostic information

======== OS Information ========
Darwin MacBook-Pro-5 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64
ProductName:            macOS
ProductVersion:         13.3.1
ProductVersionExtra:    (a)
BuildVersion:           22E772610a


======== User information ========
uid=501(tesliker) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1)


======== ddev version ========
 ITEM             VALUE                                    
 DDEV version     v1.23.0                                  
 architecture     amd64                                    
 cgo_enabled      0                                        
 db               ddev/ddev-dbserver-mariadb-10.11:v1.23.0 
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.23.0              
 docker           25.0.5                                   
 docker-api       1.44                                     
 docker-compose   v2.26.1                                  
 docker-platform  orbstack                                 
 mutagen          0.17.2                                   
 os               darwin                                   
 router           ddev/ddev-traefik-router:v1.23.0         
 web              ddev/ddev-webserver:v1.23.0-1  

======== curl -I of http://tryddevproject-8996.ddev.site from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Wed, 15 May 2024 16:49:35 GMT
Server: nginx
Vary: Accept-Encoding



======== Full curl of http://tryddevproject-8996.ddev.site from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   286    0   286    0     0  31930      0 --:--:-- --:--:-- --:--:-- 71500
Success accessing database... db via TCP/IP<br />
ddev is working.<br />
The output file for Discord or issue queue is in
<b>/var/folders/k0/y_1rz5rd1kj8d0h8pyqk04_h0000gn/T/ddev-debug-test.txt</b><br />
file:///var/folders/k0/y_1rz5rd1kj8d0h8pyqk04_h0000gn/T/ddev-debug-test.txt<br />


======== Full curl of https://tryddevproject-8996.ddev.site from outside ========
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) LibreSSL/3.3.6: error:1404B458:SSL routines:ST_CONNECT:tlsv1 unrecognized name


======== Project ownership on host ========
drwxr-xr-x@ 4 tesliker  staff  128 May 15 12:47 ../tryddevproject-8996


======== Project ownership in container ========
drwxr-xr-x 1 tesliker dialout 16 May 15 16:49 //var/www/html


======== In-container filesystem ========
Filesystem     Type  1K-blocks     Used Available Use% Mounted on
/dev/vdb1      btrfs 778501156 22373792 756127364   3% /var/www


======== Thanks for running the diagnostic! ========
Running ddev launch in 3 seconds
Running ddev launch
HTTP may redirect to HTTPS in your browser
See https://ddev.readthedocs.io/en/stable/users/usage/commands/#launch
Waiting 10 seconds to run ddev stop --unlist

Expected Behavior

Run:

ddev config
ddev start
(configure provider yml)
ddev pull

The website should be available at *.ddev.site with no errors

Actual Behavior

When I initialize a new project, I have this error:

User warning: mkdir(): Permission Denied in Drupal\Component\PhpStorage\FileStorage->createDirectory() (line 123 of core/lib/Drupal/Component/PhpStorage/FileStorage.php).
Drupal\Component\PhpStorage\FileStorage->createDirectory('sites/default/files/php', 511) (Line: 104)
Drupal\Component\PhpStorage\FileStorage->createDirectory('sites/default/files/php/twig', 511) (Line: 73)
Drupal\Component\PhpStorage\FileStorage->ensureDirectory('sites/default/files/php/twig') (Line: 66)
Drupal\Component\PhpStorage\MTimeProtectedFastFileStorage->save('657330a214eb2_file-upload-help.html.twi_JxsofamZIKhnF5cOCQHHpqEtI', '<?php

My sites/default/files folder is empty.

Once I run sudo chmod -R 777 . and restart the containers, the error goes away.

If I run sudo chmod -R 777 . before creating the project, I still get the errors and have to run it again and restart.

It is just weird that I have to do this for every project the first time. Not sure What I am doing wrong.

Steps To Reproduce

No response

Anything else?

No response

[rfay]

You shouldn't need sudo anywhere here, because the directories should already be owned by you.

if you can show ls -ld sites/default/files && ls -ld sites/default/files/* at various points during your trouble, it will help to understand this.

I assume you're using a custom hosting provider, can you share it? If not, can you say which one you're using?

With Drupal the default settings.ddev.php sets up Drupal so it doesn't try to constrain permissions. You left out a huge part of the ddev debug test (which goes to a file, you can just attach the file if you want). As a result there's a lot I can't see.

I imagine the problem is that your files tarball contains files which have limited permissions. You can easily check this by untarring the tarball in another directory and taking a look at it. If you want to share the tarball with me, I can take a look as well.

cd ~/tmp && tar -zxf <pathToTarball.tar.gz> && ls -l <directory-where-it-is>

[tesliker]

It seems this is actually a problem with the files_pull_command: in the pantheon.yaml.

So everytime I pull, the permissions are getting messed up. Not sure the right approach to fix. Adjusted the script like so, but it still does not help.

    mkdir -p files
    tar --strip-components=1 -C files -zxf files.tgz
    # Ensure the directory has the correct permissions
    sudo chmod -R 755 files
    # Ensure the directory is owned by the web server user, typically www-data in ddev containers
    sudo chown -R www-data:www-data files

[rfay]

Please use chmod ugo+rw (which makes your files writable by user and group) instead of chmod 755 (which makes all your files executable).

www-data is not who files are owned by in a DDEV setup. Where did that come from? Files in a DDEV project should be owned by your user, and mounted into the container with your user also.

It looks like you're not using https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/pantheon.yaml.example at all?

[tesliker]

Here is the full ddev debug
ddev-debug-test.txt

[tesliker]

Apologies, that was my attempt to further debug and fix the permissions issue. The code was:

files_pull_command:
  command: |
    set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null;
    terminus backup:get ${project} --element=files --to=files.tgz
    mkdir -p files && tar --strip-components=1 -C files -zxf files.tgz

[rfay]

Thanks for providing the full ddev debug test output. Everything seems fine there.

I happen to note that you don't seem to have run mkcert -install, so please do that. It has nothing to do with this issue, but you will then be able to use https URLs successfully.

[rfay]

Go ahead and use terminus inside the container to grab the files tarball (or maybe it's already there waiting for you in .ddev/.downloads) and extract it and look at the permissions it's using.

Mostly you don't want to use sudo for anything anywhere.

[rfay]

If it turns out that you can share privileges on the project so I can test against it, I'm happy to do that. Normally we only have our one simple test project. Alternately, if you want to PM me the files tarball that you can download, I'll take a look at that. I certainly haven't had this problem before.

[rfay]

Thanks for sending the tarball. It has almost nothing in it, so I assume the upstream project is almost empty?

When I untar it everything comes out appropriately and it looks fine, and none of the directories like sites/default/files/php are populated, so nothing should be the fault of the tarball.

You're not running ddev with a different user than the sites/default/files directory is owned by are you?

Do you have any trouble if you just ddev pull pantheon --skip-files and then start up?

(I don't think I've gotten you to show the output of ls -ld sites/default/files or ls -la sites/default/files yet.)

[tesliker]

Yes upstream project is almost empty. This error happens with all of the websites big and small.

ls -ld web/sites/default/files
drwxr-xr-x@ 7 tesliker  staff  224 May 15 13:32 web/sites/default/files

My user that I run ddev with is tesliker

I will try --skip-files next

[tesliker]

If I skip the files, Drupal does not throw the mkdir errors

[rfay]

And if you just --skip-db (to make things faster) what does ls -ld web/sites/default/files look like right after the pull?

(Is there any chance you have a .ddev/docker-compose.*.yaml, or have altered .ddev/mutagen/mutagen.yml?)

Does this problem go away if you ddev config --performance-mode=none ?

[rfay]

What does id -a && ddev exec id -a say?

[rfay]

Maybe you should just make an appointment and we'll look at it together with a screenshare. https://cal.com/randyfay/30min

[tesliker]

@rfay It does look like the problem went away once I turned off performance-mode.

I performed ddev pantheon pull with no issues.

I wanted to turn it back on to see if the issue happened again, but when I ran ddev pantheon pull I got this:

You're about to delete the current database and files and replace with the results of a fresh pull.
Would you like to continue? [Y/n] (yes): 
Authenticating...
 [notice] Logged in via machine token.
Obtaining databases...
+ set -eu -o pipefail
+ ls /var/www/html/.ddev
+ pushd /var/www/html/.ddev/.downloads
+ terminus backup:get agf-retiree.dev --element=db --to=db.sql.gz
 [notice] Downloading agf-retiree_dev_2024-05-15T16-30-35_UTC_database.sql.gz to db.sql.gz
Pull failed: failed to find downloaded files in /Users/tesliker/work/agf-retiree/.ddev/.downloads: <nil>

[rfay]

I'm sure you knew that you had to ddev restart when changing between mutagen and none.

My bet is if you did this a few times in a row it might be intermittent.

Someone else did have trouble with Orbstack and this situation recently, but has been impossible to sort out and unpredictable. I imagine it would be intermittent for you here as well.

Tomorrow we'll have DDEV v1.23.1, which has slightly better error reporting in situations like what you may have, so if you're able to reproduce the "nothing found in .downloads" with DDEV v1.23.1, details about it will be so very welcome.

That problem (related to what you just saw, but not related to your original problem) is a long thread in Discord: https://discord.com/channels/664580571770388500/1232668554550186024

[rfay]

Since your upstream project is so very limited, is there any possibility you could give me access to it so I could try to recreate?

Also, you say ddev pantheon pull a number of times, but I'm sure you're actually doing ddev pull pantheon, right?

[rfay]

@tesliker - I was just debugging an unrelated item and wanted to ask if you're running PhpStorm with the DDEV Plugin going. It could affect the intermittent behavior you just had.

[rfay]

Tommy and I studied this pretty carefully (thanks!) and what we found was this:

• He's using Orbstack 1.5.1 on amd64 mac
• The problem only happens when mutagen is enabled
• His ddev pull pantheon pulls a pretty trivial set of files
• After ddev pull pantheon the files directory is properly populated on the host, but it's empty inside the container, and it's not possible to mkdir inside the files dir
• ddev restart makes the files dir mount correctly and show the files on the host inside the container.

I'm able to reproduce this with our test pantheon project on Orbstack 1.5.1 also (arm64 mac). In my case, I just didn't have any twig action happening, so had never seen the problem.

Our tests on macOS + OrbStack really should be showing this behavior (TestPantheonPull should see this problem. Apparently it doesn't)

We have OrbStack 1.5.0 on tb-macos-arm64-5 and tb-macos-arm64-6, but 1.5.1 on tb-macos-arm64-7.

However, we don't do a ddev pull test on the macOS tests, only on Linux.

[rfay]

This is a bug in DDEV, at least with Drupal, possibly with other CMSs. It does not have to do with Orbstack (easy to recreate with Docker Desktop).

Here's the location of the failure:

ddev/pkg/ddevapp/drupal.go

Lines 575 to 580 in e694c20

	// If the destination path exists, remove it as was warned
	if fileutil.FileExists(destPath) {
	if err := os.RemoveAll(destPath); err != nil {
	return fmt.Errorf("failed to cleanup %s before import: %v", destPath, err)
	}
	}

What's happening is the drupalImportFilesAction() is removing the entire mounted path (sites/default/files) on the host side. But when mutagen is enabled, that is the source for the in-container files directory, and when we delete and create a new one, the old one is no longer mounted.

The correct behavior here will be to purge the target files directory, rather than removing it.

[rfay]

@tesliker could you please take a look at the artifacts in #6199 (comment) and see if they solve your problem?

You'll want the macOS-amd64 one, and you need to put it in your PATH in a place before /usr/local/bin in your case, or run it explicitly. In your situation I often put it in ~/bin and make sure ~/bin is first in PATH.

[tesliker]

@rfay Confirmed, this works. Great job, thanks!

[rfay]

Thanks for taking the time to work through this! This will be in the next release of DDEV, but of course you can use the HEAD version when this gets pulled or you also have other workarounds.