Log Directories Created After Startup not Tracked #385

stsantilena · 2022-08-03T00:32:22Z

Enables the Agent to track directories that are listed in the configuration but are initially missing. Once the directory is create its added to the file system cache and tracked.

dkhokhlov · 2022-09-10T03:11:14Z

common/fs/src/cache/dir_path.rs

+) -> Result<DirPathBuf, DirPathBufError> {
+    match path {
+        Some(p) => {
+            if p.is_dir() {


does it work for:

it is a file

permission error

The configuration is meant to be set as directory so a single file wouldn't be expected. I followed same pattern as existing directory check. As for permissions, in theory, I think could fail is for some reason it was running in non-root mode and didn't have directory access.

dkhokhlov · 2022-09-10T03:24:24Z

common/fs/src/cache/dir_path.rs

+                warn!("{:?} is not a directory; moving to parent directory", p);
+                let mut postfix_pathbuf = PathBuf::new();
+                let mut pop_pathbuf = PathBuf::new();
+                if let Some(path) = postfix {


shadowing fn arg on purpose?

The postfix function input is only non-None in the event that a given directory doesn't exist, in which case it saves that piece of the path so it can properly construct the missing directory path.

i was thinking about var name

Ah, I see. Updated var name path -> pop

common/fs/src/cache/dir_path.rs

stsantilena · 2022-09-21T21:53:12Z

common/config/src/lib.rs

@@ -344,6 +344,7 @@ impl TryFrom<RawConfig> for Config {
                // Filter off paths that are not directories and warn about them
                .filter_map(|d| {
                    d.clone()
+                        //TODO: Need to fix this iteration as warning now happens in dir_path.rs


@c-nixon I don't think there needs to any change here, perhaps the warning might be different message? Also, thinking the iteration structure should stay for Windows.

if we allow to specify non-existing dirs - why do we filter them here out?

This iteration just filters out anything that throws an error. Originally, if a directory didn't exist and error was thrown. Now, there is a more complex process happening down in find_valid_path that allows directories that don't yet exist; still propagating errors up but for different reasons. So really just the comment is no longer valid.

stsantilena · 2022-09-21T21:55:54Z

common/fs/src/cache/dir_path.rs

-            Ok(DirPathBuf { inner: path })
+            Ok(DirPathBuf {
+                inner: path,
+                postfix: None,


Had to add this line to fix the Windows build. This shouldn't affect existing Windows behavior.

stsantilena · 2022-09-21T23:06:38Z

bin/tests/it/cli.rs

@@ -1573,7 +1573,6 @@ async fn test_line_redact() {
    test_line_rules(None, None, redact, to_write, expected).await;
 }

-#[ignore]
 #[tokio::test]
 async fn test_directory_created_after_initialization() {


Turned on bug specific integration test from previous PR.

stsantilena · 2022-09-26T23:52:11Z

common/fs/src/cache/dir_path.rs

+                let parent = match level_up(&p) {
+                    Some(p) => {
+                        if p == root_pathbuf {
+                            panic!("configured directory recursed to root!");


Set to panic if function recurses back to the root directory. This will watch the entire root file system which is most likely not desirable.

Changed this so that instead of throwing a panic, it will warn and watch root level for missing dir.

c-nixon

This all looks good to me!

c-nixon · 2022-10-04T14:46:41Z

common/fs/src/cache/dir_path.rs

+            } else {
+                warn!("{:?} is not a directory; moving to parent directory", p);
+                let root_pathbuf = Path::new("/");
+                let mut postfix_pathbuf = PathBuf::new();


I feel like there might be some way to use the std::path Component types to implement this, but that might just complicate the logic

There's probably a few different ways to go about it. The tricky part is flipping the path each integration to make sure it comes out correct, and plays nicely with the path rules.

stsantilena added bug Something isn't working enhancement New feature or request labels Aug 17, 2022

stsantilena requested a review from c-nixon August 18, 2022 20:27

stsantilena removed the enhancement New feature or request label Aug 18, 2022

stsantilena changed the title ~~[WIP] Log Directories Created After Startup not Tracked~~ Log Directories Created After Startup not Tracked Sep 9, 2022

stsantilena marked this pull request as ready for review September 9, 2022 22:24

stsantilena requested review from dkhokhlov and james-t-bloom September 9, 2022 22:25

dkhokhlov reviewed Sep 10, 2022

View reviewed changes

stsantilena commented Sep 21, 2022

View reviewed changes

stsantilena commented Sep 26, 2022

View reviewed changes

c-nixon approved these changes Oct 4, 2022

View reviewed changes

c-nixon reviewed Oct 4, 2022

View reviewed changes

stsantilena added 14 commits October 7, 2022 12:55

Run log dir integration test

00545ab

Added parent directory logic

444313a

Change error handling and unit test

18bce35

Modified recursive path backup

7db6a7c

Fix lint errors

0a48057

Refactor helper function

91b24d5

Adding missing dirs object to mod

5f36a45

Add missing dirs watcher

ea50e41

Started adding streaming logic

d639787

Fix dir path algo

2fbe3d0

Move stream to stream events

34fde03

Refactored stream unfold

3d002c3

Fix watcher addition

19dbb22

Adding missing dir rules

47451f5

stsantilena added 12 commits October 7, 2022 12:55

Fix sub dir create event and clean up

20641c5

Fix integration tests

6e582bf

Fix windows build error

bbde603

Fix logging typo

e70289a

Activated integration test

309659e

Added additional unit tests

0fc32ae

Changed var name

330f798

Fixed root level watcher

d104f9d

Fix lint errors

c28fc85

Add additional unit tests

6ec9b53

Address PR comments and update find valid path function

cdf5843

Simplify logical expression

fb32617

c-nixon force-pushed the LOG-9806-log-dir-not-tracked-cont branch from 909d8e8 to fb32617 Compare October 7, 2022 10:55

c-nixon merged commit 64ab339 into master Oct 7, 2022

c-nixon deleted the LOG-9806-log-dir-not-tracked-cont branch October 7, 2022 14:09

stsantilena mentioned this pull request Nov 10, 2022

Fix Windows Compiler Warnings #448

Merged

dkhokhlov linked an issue Jan 26, 2023 that may be closed by this pull request

Agent monitoring directories that dont existing at start time. #411

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log Directories Created After Startup not Tracked #385

Log Directories Created After Startup not Tracked #385

stsantilena commented Aug 3, 2022 •

edited

dkhokhlov Sep 10, 2022

stsantilena Sep 20, 2022

dkhokhlov Sep 10, 2022

stsantilena Sep 19, 2022

dkhokhlov Sep 22, 2022

stsantilena Sep 27, 2022

stsantilena Sep 21, 2022 •

edited

dkhokhlov Sep 29, 2022 •

edited

stsantilena Sep 29, 2022

stsantilena Sep 21, 2022

stsantilena Sep 21, 2022

stsantilena Sep 26, 2022

stsantilena Sep 27, 2022

c-nixon left a comment

c-nixon Oct 4, 2022

stsantilena Oct 4, 2022

Log Directories Created After Startup not Tracked #385

Log Directories Created After Startup not Tracked #385

Conversation

stsantilena commented Aug 3, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsantilena Sep 21, 2022 • edited

Choose a reason for hiding this comment

dkhokhlov Sep 29, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c-nixon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsantilena commented Aug 3, 2022 •

edited

stsantilena Sep 21, 2022 •

edited

dkhokhlov Sep 29, 2022 •

edited