New feature: capture image #93
Conversation
* uses new command added to index.js * takes the b64 encoded dataURI from websocket, sends bytes over zmq
* makes use of queue.Queue pointless, removed * also removed method ws_receive_image_process_data()
rdeits
left a comment
There was a problem hiding this comment.
This will certainly be useful, thank you!
src/meshcat/commands.py
Outdated
| class CaptureImage: | ||
| __slots__ = ["save_path"] | ||
| def __init__(self, save_path=""): | ||
| self.save_path = save_path |
There was a problem hiding this comment.
It seems like save_path isn't actually used anywhere, since the image is returned as a PIL.Image (a good idea) rather than being written to disk. Can we just remove save_path entirely?
There was a problem hiding this comment.
Yes, let's remove it.
There was a problem hiding this comment.
Commit c7a4c73 adresses this, though I had to insert a UTF-8 encoded empty string to make sure the length of the frames arg is right
| self.zmq_socket.send_multipart([ | ||
| cmd_data["type"].encode("utf-8"), | ||
| cmd_data["save_path"].encode("utf-8"), | ||
| "".encode("utf-8"), |
There was a problem hiding this comment.
This is what I was talking about. This makes sure frames in ZMQSocketBridge.forward_to_websockets has the right length.
Ideally we could change that method to be more flexible, too
There was a problem hiding this comment.
Ok, makes sense. I think this is fine for now.
src/meshcat/servers/zmqserver.py
Outdated
| elif cmd == "wait": | ||
| self.ioloop.add_callback(self.wait_for_websockets) | ||
| elif cmd == "capture_image": | ||
| save_path = frames[1].decode("utf-8") |
There was a problem hiding this comment.
This line was actually useless so I removed it
| "tornado >= 4.0.0", | ||
| "pyzmq >= 17.0.0", | ||
| "pyngrok >= 4.1.6", | ||
| "pillow >= 7.0.0" |
There was a problem hiding this comment.
I added the requirement for PIL
src/meshcat/visualizer.py
Outdated
| vis.path = path | ||
| return vis | ||
|
|
||
| viewer = meshcat.Visualizer() |
There was a problem hiding this comment.
this line failed the linting, don't know where it came from
|
@rdeits Can we get a new release on PyPI after merging? |
|
Yup, I'll do that as soon as this is merged. |
Codecov Report
@@ Coverage Diff @@
## master #93 +/- ##
==========================================
- Coverage 74.85% 73.52% -1.33%
==========================================
Files 11 11
Lines 1006 1035 +29
==========================================
+ Hits 753 761 +8
- Misses 253 274 +21
Continue to review full report at Codecov.
|
Added upstream in meshcat-dev/meshcat-python#93
This PR follows meshcat-dev/meshcat#85.
It adds a new
get_image()method tomeshcat.Visualizer. This relies on the newcapture_imagecommand added in meshcat-dev/meshcat#85 which queries an image returned to the Meshcat client as a PNG in base64 data URI form.Design
On the Python side, the WebsocketHandler awaits the response from the Meshcat client, and when it receives the JSON from the frontend, the
on_message()callback is triggered, which sends the PNG (turned into a sequence of bytes) over the ZMQ bridge.Limitations
The design is not completely robust yet. It requires the browser window be already open to get a response in the client (the visualizer will block until then).
Maybe this behavior can be fixed by using something else to get the PNG render.