Adds limit parameter to ansible.builtin.find

[colin-nolan]

SUMMARY

ansible.builtin.find currently looks for matches in all non-filtered files. When there is a very big file system in the search scope, and/or an expensive search is made (e.g. based off whole file contents), the operation can be very expensive.

This PR adds the ability for the user to limit the number of matches that are made. Use cases that would benefit/be enabled with this functionality include:

• Determining if there are > n matching files in a directory.
• Finding the first directory in a nested tree with any files.
• Discovering where the file with matching content is (stopping once found!).

ISSUE TYPE

• Feature Pull Request

ADDITIONAL INFORMATION

Try the module using:

# Can't run module in place due to import issues caused by other modules in directory
cp lib/ansible/modules/find.py /tmp/find.py && python /tmp/find.py <<EOF
{
    "ANSIBLE_MODULE_ARGS": {
        "paths": "/var/log",
        "file_type": "file",
        "contains": "wally",
        "read_whole_file": true,
        "patterns": "^.*\\\.log$",
        "use_regex": true,
        "recurse": true,
        "limit": 1
    }
}
EOF

[colin-nolan]

I appreciate you likely have more important work to deal with but tagging module author @bcoca, as it feels like ansibot should have.

[bcoca]

I'm not even sure this feature is needed as it is a simple thing to cut down the return of files to only those you need returned_files[:5].

Though this can be a performance improvement, it would only be really noticeable with huge amounts of files.

[bcoca]

use None for 'unset/unlimited'

[colin-nolan]

Will change. Am I correct in thinking that the type should still be int, given there's no Optional[int] type (https://docs.ansible.com/ansible/latest/dev_guide/developing_program_flow_modules.html#argument-spec)?

[colin-nolan]

Have changed. I'm not sure what the correct value to set as default in the docs should be now that it's None (see test failure). Any help would be appreciated.

[bcoca]

you can just remove it, None is the default for default. All types allow None as a default as it is 'unset'.

[bcoca]

i would rename limit or 'maximum` as this does not relate to a 'match' field, but to all the conditions the other options can set

[colin-nolan]

Re-named to limit.

[bcoca]

more like:
The maximum number of matching paths returned, after finding this many the find action will stop looking.

[colin-nolan]

Updated

[bcoca]

this changes the normal search by adding topdown, this is not really backwards compatible

[bcoca]

i would even add this as an option before i limit matching

[colin-nolan]

topdown is True by default (https://docs.python.org/3/library/os.html#os.walk) so it should not change the current behaviour. The intention was to make it explicit that the module is guaranteed to be topdown - perhaps the docs is a better place for this though?

I'm not sure how many people will have a use-case where they'd want to switch between topdown/bottomup. However, I'm happy to add it if you think it could be useful.

[ansibot]

The test ansible-test sanity --test validate-modules [explain] failed with 2 errors:

lib/ansible/modules/find.py:0:0: doc-default-incompatible-type: Argument 'limit' in documentation defines default as ('None') but this is incompatible with parameter type 'int'
lib/ansible/modules/find.py:0:0: incompatible-default-type: DOCUMENTATION.options.limit: Argument defines default as ('None') but this is incompatible with parameter type int: invalid literal for int() with base 10: 'None' for dictionary value @ data['options']['limit']. Got {'description': ['Limit the maximum number of matching paths returned. After finding this many, the find action will stop looking.', 'Matches are made from the top, down (i.e. shallowest directory first).', 'Set to V(None) for unlimited matches.', 'Default is unlimited matches.'], 'type': 'int', 'default': 'None', 'version_added': '2.18', 'version_added_collection': 'ansible.builtin'}

click here for bot help

[colin-nolan]

@bcoca many thanks for reviewing. I have addressed most of your comments. Just two things left to resolve:

1. How to document a default of None, where None is not of type int (currently failing a check).
2. Whether you think an option to change between topdown/bottom up is required. I'd personally suggest putting it out of scope, as it's difficult to think of many use-cases where it would be useful.

I'm not even sure this feature is needed as it is a simple thing to cut down the return of files to only those you need returned_files[:5].

Though this can be a performance improvement, it would only be really noticeable with huge amounts of files.

This is the exact situation that I have - lot's of files, some of them very large. The module isn't really viable for my use-case without this option.