-
-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use WinAPI to simulate pointermovements #128
base: master
Are you sure you want to change the base?
Conversation
Hmm, how does captrs work, it looks like you can pass some kind of ID to new: https://docs.rs/captrs/0.3.1/captrs/struct.Capturer.html#method.new, how does it map to monitors? Regarding your implementation, it's better not to modify Lines 10 to 16 in e23d3b9
Lines 513 to 517 in e23d3b9
|
Yep the id is the monitor nr. 0 is the primary
yep will do that this was just for fast testing |
That's nice, implementing a capturable should be straight forward then. There is a Capturable trait: Lines 33 to 47 in e23d3b9
and a Recorder trait Lines 16 to 18 in e23d3b9
that need to be implemented. Have a look at https://github.com/H-M-H/Weylus/blob/master/src/capturable/autopilot.rs for the most basic Capturable + Recorder. You could add a new captrs capturable that stores its monitor id. As captrs returns a &[] to #[repr(C)]
pub struct Bgr8 {
pub b: u8,
pub g: u8,
pub r: u8,
_pad: u8,
} (using Line 55 in e23d3b9
Alright! |
My capturer implementation doesnt compile. Dont really know what to do about that. Its my first time really using rust ... use crate::capturable::{Capturable, Recorder};
use captrs::Capturer;
use std::boxed::Box;
use std::error::Error;
#[derive(Clone)]
pub struct CaptrsCapturable {
id: u8,
capturer: Capturer,
}
impl CaptrsCapturable {
pub fn new(id: u8) -> CaptrsCapturable {
let capturer = Capturer::new(id).unwrap();
CaptrsCapturable { id, capturer }
}
}
impl Capturable for CaptrsCapturable {
fn name(&self) -> String {
"Desktop (captrs)".into()
}
fn geometry_relative(&self) -> Result<(f64, f64, f64, f64), Box<dyn Error>> {
Ok((0.0, 0.0, 1.0, 1.0))
}
fn before_input(&mut self) -> Result<(), Box<dyn Error>> {
Ok(())
}
fn recorder(&self, _capture_cursor: bool) -> Result<Box<dyn Recorder>, Box<dyn Error>> {
Ok(Box::new(CaptrsRecorder::new()))
}
}
pub struct CaptrsRecorder {
img: Vec<u8>,
}
impl CaptrsRecorder {
pub fn new() -> CaptrsRecorder {
CaptrsRecorder { img: Vec::new() }
}
}
impl Recorder for CaptrsRecorder {
fn capture(&mut self) -> Result<crate::video::PixelProvider, Box<dyn Error>> {
self.capturer.capture_store_frame();
let (w, h) = self.capturer.geometry();
self.img = self.capturer.get_stored_frame().unwrap();
Ok(crate::video::PixelProvider::BGR0(w, h, &self.img))
}
}
|
Don't worry :) The Capturable + Recorder traits are designed in such a way that the Capturable can be safely copied and shared among threads. So a Capturable should only store minimal meta information about what it is (in this case which display) and actually capturing the screen should be delegated to the Recorder which doesn't need to be cloneable or thread safe. The concrete problem you have run into is that captrs Capturer is not thread safe. This means it "poisons" your CaptrsCapturable struct. The solution is to just store the display ID inside the Capturable and instantiate the Capturer only with the Recorder. I hope this clears things up, gotta get some sleep now, so I will take some time to reply if something comes up. |
the last thing missing now is a way to get the screensize from a capturable to calculate the coordinates. I thought about adding a screen_size function to the capturable trait but that wont work since in my case the recorder holds the captrs variable that has the screensize. |
Looks like captrs uses In the longrun I'd like to get rid of third party crates for screen capturing though as non of those (at least to my knowledge) provide a way to capture individual windows which is something I'd like to see supported on all platforms (actually only the Windows implementation is missing). But that's probably a significant amount of work and I understand if you are not up for it. |
Another thing to look at is the coordinate offset. Sadly captrs implements a function to get the offset for linux but not for windows. The best thing i could get is this via EnumDisplayMonitors and GetMonitorInfo:
Calculating the offsets is weird though since: I have a working setup for 1 monitor though :) Btw this doesnt return the same order as the EnumMonitorDevices method. Main Display is 1 with EnumMonitorDevices and 0 with EnumDisplayDevicesA for whatever reason
|
Oh well, I looked into the docs and saw a But the good news is that it looks like directly calling winapi can solve this:
I wouldn't go there, that indeed sounds like different layouts would immediately break this.
I'd say getting
Awesome, that's great progress! |
You are right that using EnumDisplays will probably be the easiest way to get the RECT of each Monitor indexed in a way that captrs will understand. Thats not really my problem... Sorry if i failed to communicate that in my previous reply, i was in a rush when writing that.
For InjectSyntheticPointerInput this is not the case though. MSDOC: "The ptPixelLocation for each POINTER_TYPE_INFO is specified relative to top left of the virtual screen" So (0,0) will end up to be the top-left corner of Monitor2 in the picture above. What im struggeling to wrap my head around is an easy way to calculate the actual coordinate of the top-left corner of each monitor on the virtualdisplay-coordinatesystem as its understood by InjectSyntheticPointerInput |
Oh, I see. If there is indeed some kind of offset the following should be correct: The coordinate systems only differ by a translation, so to go from getDesc to Virtual Screen you just need to subtract the minimal left value over all monitors as well as the minimal top value respectively for left and right, top and bottom of the monitor coordinates you want to convert. |
Yes i think thats right, i was testing just now and came up with this: void print() {
int i = 0;
for (const RECT &r : this->rcMonitors) {
std::printf("Monitor[%i] Left: %i Top: %i Right: %i Bottom: %i\n", i++, r.left, r.top, r.right,r.bottom);
}
std::printf("Union: Left: %i Top: %i Right: %i Bottom: %i\n", this->rcCombined.left , this->rcCombined.top, this->rcCombined.right, this->rcCombined.bottom );
std::printf("\n\n");
for (const RECT& r : this->rcMonitors) {
std::printf("Monitor+Offset[%i] Left: %i Top: %i Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top, r.right + rcCombined.right, r.bottom);
}
}
|
That's looking good, it appears top and bottom behave as I expected. I do think instead of std::printf("Monitor+Offset[%i] Left: %i Top: %i Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top, r.right + rcCombined.right, r.bottom); it should be std::printf("Monitor+Offset[%i] Left: %i Top: %i Right: %i Bottom: %i \n", i++,r.left - rcCombined.left, r.top - rcCombined.top, r.right - rcCombined.left, r.bottom - rcCombined.top); though. |
Ok cool i think we have this worked out. use std::mem::zeroed;
use std::{mem, ptr};
use winapi::shared::dxgi::{
CreateDXGIFactory1, IDXGIAdapter1, IDXGIFactory1, IDXGIOutput, IID_IDXGIFactory1,
DXGI_OUTPUT_DESC,
};
use winapi::shared::windef::*;
use winapi::shared::winerror::*;
use winapi::um::winuser::*;
use wio::com::ComPtr;
// from https://github.com/bryal/dxgcap-rs/blob/009b746d1c19c4c10921dd469eaee483db6aa002/src/lib.r
fn hr_failed(hr: HRESULT) -> bool {
hr < 0
}
fn create_dxgi_factory_1() -> ComPtr<IDXGIFactory1> {
unsafe {
let mut factory = ptr::null_mut();
let hr = CreateDXGIFactory1(&IID_IDXGIFactory1, &mut factory);
if hr_failed(hr) {
panic!("Failed to create DXGIFactory1, {:x}", hr)
} else {
ComPtr::from_raw(factory as *mut IDXGIFactory1)
}
}
}
fn get_adapter_outputs(adapter: &IDXGIAdapter1) -> Vec<ComPtr<IDXGIOutput>> {
let mut outputs = Vec::new();
for i in 0.. {
unsafe {
let mut output = ptr::null_mut();
if hr_failed(adapter.EnumOutputs(i, &mut output)) {
break;
} else {
let mut out_desc = zeroed();
(*output).GetDesc(&mut out_desc);
if out_desc.AttachedToDesktop != 0 {
outputs.push(ComPtr::from_raw(output))
} else {
break;
}
}
}
}
outputs
}
pub struct win_ctx {
outputs: Vec<RECT>,
union_rect: RECT,
}
impl win_ctx {
pub fn new() -> win_ctx {
let mut rects: Vec<RECT> = Vec::new();
let mut union: RECT;
unsafe {
union = mem::zeroed();
let factory = create_dxgi_factory_1();
let mut adapter = ptr::null_mut();
if factory.EnumAdapters1(0, &mut adapter) != DXGI_ERROR_NOT_FOUND {
let adp = ComPtr::from_raw(adapter);
let outputs = get_adapter_outputs(&adp);
for o in outputs {
let mut desc: DXGI_OUTPUT_DESC = mem::zeroed();
o.GetDesc(ptr::addr_of_mut!(desc));
rects.push(desc.DesktopCoordinates);
UnionRect(
ptr::addr_of_mut!(union),
ptr::addr_of!(union),
ptr::addr_of!(desc.DesktopCoordinates),
);
}
}
}
win_ctx {
outputs: rects,
union_rect: union,
}
}
pub fn get_outputs(&self) -> &Vec<RECT> {
&self.outputs
}
pub fn get_union_rect(&self) -> &RECT {
&self.union_rect
}
} What would be the best way to get the current screen_width, screen_height, offset_x, offset_y back to the InputDevice? |
Looks like the input device needs to know about the window context too, so I guess it can't be helped but to lookup the dimensions of the Virtual Screen inside the input device with its own handle as well. Then the Capturables Perhaps a little explanation why I designed things like this: On Linux you can simulate inputs on a pretty low level with absolutely no knowledge of display sizes or positions. You can even declare your own (rectangular) input coordinate system which on some higher level is then mapped to actual screen coordinates. |
It works now with multiple monitors, i edited the capturable trait though. I dont think thats the perfect solution, but i wanted to see if the coordinate translation is correct before figuring the rest out Just tell me what i have to change for it to get merged and i will try my best :) |
I'd say lets get it building everywhere, fixup the trait and then I will have a look again. Thanks for your work! |
Ragarding fixing up the trait. While researching the Box pattern i found this post and if i understand it correctly this fits here. Since all of the Capturable implementations hold different data, that might be needed in the implementations of Input but there is no way to get back to the original struct after putting it in a box. |
How about a compromise, I still think enum Geometry {
Relative(f64, f64, f64, f64),
VirtualScreen(i32, i32, u32, u32),
} The VirtualScreen variant can then be marked |
Sounds good |
Any progress regarding touch input? Can I help somehow? |
Sry dont have too much spare time at the moment. The only thing missing regarding touch input is correct handling of multitouch. I do get multiple pointers for multiple fingers but they dont work together. Another thing i had a deeper look at was this. I was able to confirm that window-capturing is possible like its done there, but i wasnt able to integrate it with weylus yet. Multitouch looks like this atm: |
Thanks for the update!
Nothing to worry about, I mean we all do this in our spare time. Also have a merry Christmas 🎄! |
Thank you 😃, have nice holidays aswell 🎅 |
First of all happy new year! :) Regarding display and window capture again. I asked the author of displayrecorderand screenshot-rs if its is possible to get a unencoded frame the way hes capturing in displayrecorder here. The answer was sadly, no. But since hes using mediafoundation aswell maybe this could still be usefull. I just donk know enough about directx and mediafoundation to judge that. |
@qdlmcfresh how's this going, will you have time in the foreseeable future? If not, perhaps somebody else using Windows as well is willing to take this over and get multi touch working. |
Time isnt even the issue here. I just dont know how to solve the problem with multitouch ... So if anybody has something to add to this they are very welcome to do so. :) |
#156 fixed the issue. |
Since @Ph-Fr-One is no longer with us i took the liberty to integrate his changes into this PR. I tested with 2 screens on Win10 and zoom/pinch gestures are working fine. |
I've seen that captrs is constantly reporting Are these just some expected and okay-to-ignore errors when working with captrs, or do I need some further configurations? Update: it turns out to live happily with these errors, so I think I could safely ignore them. |
Excuse me, any problem with this PR? @H-M-H |
Hello hello guys i'v just read your topic and it soo cool, but im little pickle with all does stuff and I don't understand any of it. i'm trying to understand it but its hard. So did you finally manage to add pointer movment with weylus ? I would really love to use it if you could show me how to install it. Cause I got now but it does not work with pen sensivity of my Ipad :/ |
https://github.com/kotaewing/Weylus/releases/tag/V1.1 should be a build with the changes in this PR integrated. |
Wow thats super man thanks ! Now the only thing would be to add a wired
connection to it so there will be no latency. Its like almost the perfect
software but trying to sketch on Photoshop with it is a bit tricky with the
delay
Le jeu. 2 mars 2023, 03 h 44, qdlmcfresh ***@***.***> a
écrit :
… Hello hello guys i'v just read your topic and it soo cool, but im little
pickle with all does stuff and I don't understand any of it. i'm trying to
understand it but its hard. So did you finally manage to add pointer
movment with weylus ? I would really love to use it if you could show me
how to install it. Cause I got now but it does not work with pen sensivity
of my Ipad :/
https://github.com/kotaewing/Weylus/releases/tag/V1.1 should be a build
with the changes in this PR integrated.
—
Reply to this email directly, view it on GitHub
<#128 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/A6G24TG2SIQDM3G7IJCNPVLW2BMXZANCNFSM5IZEFYBQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I got this to build with the hhmhh/weylus_build docker image by
Also for weylus.exe to work I had to bundle it with all the dll files from the bin folders from those zips. Very cool this works. This message is constantly spammed in the log, but I didn't notice an impact: |
Does this work for anyone with server = windows, client = linux? If not I bet it'd be pretty close based on this work so far. For me I can do windows to windows, but when I try remoting to a windows machine from linux, the stylus events don't show up on the windows side. And I can confirm with This would be super awesome to have a weylus server in a windows VM and be able to use the stylus from a a linux host. I don't think there's any other solution right now to do linux -> windows stylus stuff. If anyone knows of anything I'd love to hear. For windows to windows, RDP already supports pen & pressure and touch. But this isn't implemented in freerdp yet FreeRDP/FreeRDP#7260 |
I also just tested that windows <--> windows RDP shadowing work with the pen and pressure too. Shadowing is screen is mirrored on both sides without RDP taking over, so very similar to this PR but built into windows. Also unfortunately this is not implemented in freerdp for linux --> windows FreeRDP/FreeRDP#5656 To try the rdp shadowing, see group policy for Computer Configuration -> Administrative Templates -> Windows components -> Remote Desktop Services -> Remote Session Host -> Connections -> Set rules for remote control of Remote Desktop Services user sessions and then
Oh, with the limiting factor of needing Pro editions of windows which sucks. I'll try to look at the linux --> windows possibilities with this PR later |
Add pen support for #17
Touch is not implemented yet and at the current stage i think it will only compile on windows.
Since the coordinates sent by the event are not padded to fit the actual monitorlayout on a multimonitor setup, the pointer will allways be on the leftmost monitor.
The library captrs supports taking screenshots of multiple monitors, so this could be used instead of autopilot.
To get the number of connected and active displays this can be used.