So you’re happily tweeting away as your Google self-driving car crosses a bridge, its speed precisely synced to the 50 m.p.h. limit. A group of frisky schoolchildren is also heading across the bridge, on the pedestrian walkway. Suddenly, there’s a tussle, and three of the kids are pushed into the road, right in your vehicle’s path. Your self-driving car has a fraction of a second to make a choice: Either it swerves off the bridge, possibly killing you, or it runs over the children. What does the Google algorithm tell it to do?<\/p>\n
This is the type of scenario that NYU psychology professor Gary Marcus considers as he ponders<\/a> the rapid approach of\u00a0a time when “it will no longer be optional for machines to have ethical systems.” As we begin to have computer-controlled cars, robots, and other machines operating autonomously out in the chaotic human world, situations will inevitably arise in which the software has to choose between a set of bad, even horrible, alternatives. How do you program a computer to choose the lesser of two evils? What are the criteria, and how do you weigh them? Since we humans aren’t very good at codifying responses to moral dilemmas ourselves, particularly when the precise contours of a dilemma can’t be predicted ahead of its occurrence, programmers will find themselves in an extraordinarily difficult situation. And one assumes that they will carry a moral, not to mention a legal, burden for the code they write.<\/p>\n The military, which already operates automated killing machines, will likely be the first to struggle in earnest with the problem. Indeed, as Spencer Ackerman noted<\/a> yesterday, the U.S. Department of Defense has just issued a directive<\/a>\u00a0that establishes rules “designed to minimize the probability and consequences of failures in autonomous and semi-autonomous weapon systems that could lead to unintended engagements.” One thing the Pentagon hopes to ensure is that, when autonomous weapons use force, “appropriate levels of human judgment” are incorporated into the decisions.\u00a0But nowhere is the world more chaotic than in a war zone, and as fighting machines gain more sophistication and autonomy and are given more responsibility, “unintended engagements” will happen. Barring some major shift in strategy, a military robot or drone will eventually be in an ambiguous situation and have to make a split-second decision with lethal consequences. Shoot, or hold fire?<\/p>\n We don’t even really know what a conscience is, but somebody’s going to have to program one nonetheless.<\/p>\n","protected":false},"excerpt":{"rendered":" So you’re happily tweeting away as your Google self-driving car crosses a bridge, its speed precisely synced to the 50 m.p.h. limit. A group of frisky schoolchildren is also heading across the bridge, on the pedestrian walkway. Suddenly, there’s a tussle, and three of the kids are pushed into the road, right in your vehicle’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-2173","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/posts\/2173","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2173"}],"version-history":[{"count":10,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/posts\/2173\/revisions"}],"predecessor-version":[{"id":2176,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=\/wp\/v2\/posts\/2173\/revisions\/2176"}],"wp:attachment":[{"href":"https:\/\/www.roughtype.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2173"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2173"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.roughtype.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2173"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}