{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Downward Camera Analysis + Line Detection\n", "In this lesson, you will learn about how to analyze the output of the downward-facing camera and use it in your code development for line following.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Integrating `opencv` and ROS images\n", "You will now learn how to use ROS images with `opencv`. The key package you will be using to do so is `cv_bridge`. \n", "\n", "The following code is a skeleton node that you will need to complete. In your finished program, you should be able to subscribe to the downward camera and publish a processed image that you should be able to see/verify in `rqt_image_view`\n", "\n", "Copy-paste the following code into a python file and work from there. You will write this program **as a team, on the drone**. \n", "\n", "```python\n", "import cv2 \n", "from cv_bridge import CvBridge, CvBridgeError\n", "import rospy\n", "from sensor_msgs.msg import Image\n", "\n", "\"\"\"\n", "Starter code for an image processing node. Subscribes to the downward camera\n", "and publishes a processed cv image (i.e. colorized, overlayed with a detected line)\n", "This node is not meant to be a full node that you will use in your final line tracking\n", "solution, but it will teach you key concepts that you will need for line following\n", "\"\"\"\n", "\n", "class ImageToCV:\n", " def __init__(self):\n", " # A subscriber to the topic '/aero_downward_camera/image'. self.image_sub_cb is called when a message is recieved \n", " self.camera_sub = rospy.Subscriber(\"/aero_downward_camera/image\", Image, self.image_cb)\n", "\n", " # A publisher which will publish a parametrization of the detected line to the topic '/image_to_cv/processed'\n", " self.image_pub = rospy.Publisher(\"/image_to_cv/processed\", Image, queue_size=1)\n", " self.bridge = CvBridge()\n", " \n", " def image_cb(self, msg):\n", " \"\"\"\n", " Callback function which is called when a new message of type Image is recieved by self.camera_sub.\n", " \n", " Args: \n", " - msg = ROS Image message in 8UC1 format\n", " \n", " Returns: NA\n", " \"\"\"\n", " try:\n", " cv_image = self.bridge.imgmsg_to_cv2(msg, \"8UC1\")\n", " self.process(cv_image)\n", " except CvBridgeError as e:\n", " print(e)\n", " \n", " def process(self, img):\n", " \"\"\"\n", " Takes in an img param and finds the line in it, if it exists\n", " \n", " Args: \n", " - img : OpenCV Image to process\n", " \n", " Publishes:\n", " - annotated image to the /image_to_cv/processed topic\n", " \"\"\"\n", " \"\"\"TODO-START: FILL IN CODE HERE\"\"\"\n", " # Get a colored copy of the image. This will be used solely to provide an annotated version\n", " # of the image for debuging purposes (for you to see in rqt_image_view). See slides for help here\n", "\n", " # Threshold the image to isolate the line and use it for your linear regression algorithm\n", " # (tip: check size to ensure you are detecting the LED line or a simple blip on the screen)\n", "\n", " # \"Draw\" a line if you have found one on the image\n", "\n", " # Publish your newly annotated, color image\n", " # NOTE: If you do not detect a line, or if you have decided to ignore detected lines of very small\n", " # size, be sure to still publish the image. Also be sure to have some way to handle the camera\n", " # seeing multiple \"lines\" at one time\n", "\n", "\n", " \"\"\"TODO-END\"\"\"\n", " \n", "if __name__ == \"__main__\":\n", " rospy.init_node(\"image_to_cv\")\n", " a = ImageToCV()\n", " \n", " rospy.spin()\n", "```\n", "\n", "### Visualization\n", "When you have finished writing the program, make sure you make it exectuable (you only need to do this once) using the usual command (`chmod +x filename.py`). Do the following to launch your code:\n", "Terminal 1:\n", "```\n", "roscore\n", "```\n", "Terminal 2 (starts optical flow):\n", "```\n", "sudo -E bwsi-uav/catkin_ws/src/aero-optical-flow/build/aero-optical-flow\n", "```\n", "Terminal 3 (run your code):\n", "```\n", "python filename.py\n", "```\n", "Terminal 4:\n", "\n", "> Use `rqt_image_view` to visualize the resulting images on `/image_to_cv/processed`. If all goes well, you should see an annotated image when viewing this topic in `rqt_image_view`, with a contour or a line over where your detected line is.\n", "(*For reference:* your camera feed should look a lot like the images you worked with in the Contour Analysis section of the Downward.ipynb lab.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intro to `rosbag`s" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`rosbag` is an ROS tool that allows you to record the output of any number of ROS topics while the drone is in operation. This tool is useful for code development because it means we can test without having a drone flying around all the time. We will be using it to look at the LED strip you will have to follow in this week's challenge. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Syntax of rosbag record\n", "`rosbag record [options] / [/...]`\n", "\n", "- Usually, you should use `-O .bag` for your `[option]`. The `-O` (capital O) flag tells ros that the following argument (formatted `.bag`) is the name we want for our bag file. Otherwise, a default, messy name will be used.\n", "\n", "- For this lab's purpose, you should use this topic for `/`: `/aero_downward_camera/image`\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Recording a `rosbag`\n", "**SSH into the drone.**\n", "\n", "Terminal 1:\n", "```\n", "roscore\n", "```\n", "Terminal 2 (starts optical flow):\n", "```\n", "sudo -E bwsi-uav/catkin_ws/src/aero-optical-flow/build/aero-optical-flow\n", "```\n", "Terminal 3:\n", "```\n", "cd ~/rosbags\n", "``` \n", "> (if the `rosbags` directory doesn't exist, run `mkdir ~/rosbags` and then `cd ~/rosbags`)\n", "\n", "Now, you will record a bag file. Make sure to use `rqt_image_view` on the drone to see what you are recording as you record it. Keep in mind that you will be recording a lot of data in your bag file, even if you are recording only one topic. We will learn later how to record compressed bags, but for now, just remember to **keep your bag recordings relatively short, around 30 seconds maximum** to avoid filling up your storage.\n", "\n", "Use this command for today's lab:\n", "```\n", "rosbag record -O downward.bag /aero_downward_camera/image\n", "```\n", "\n", "When you are finished recording, `ctrl-c` in the window you are recording your bag in to stop it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting your bag off the drone for analysis\n", "After recording a bag, especially a long one, your drone storage might be relatively full. If you want, check disk usage with `sudo df -h /`.\n", "We will now transfer the bag file from the drone to your laptop and delete the bag file from the drone to free up space.\n", "\n", "**On your laptop:**\n", "```\n", "cd ~/Desktop (or other dir in which you want the bagfile to go)\n", "scp uav@.beaver.works:~/rosbags/downward.bag ~//.\n", "```\n", "**On the drone:**\n", "```\n", "cd ~/rosbags\n", "rm downward.bag\n", "```\n", "\n", "Your bag should now be on your laptop and deleted off the drone. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Playing your bag file\n", "**On your laptop:**\n", "\n", "Terminal 1:\n", "```\n", "roscore\n", "```\n", "Terminal 2:\n", "\n", "> In the folder where you saved your bag file, use `rosbag play .bag` to \"play\" the bag file.\n", "\n", "Terminal 3:\n", "```\n", "rqt_image_view\n", "```\n", "If you have completed the above code correctly, you should see an annotated image when you select the `/aero_downward_camera/image` in `rqt_image_view`." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }