{"id":59805,"date":"2021-01-21T09:13:48","date_gmt":"2021-01-21T00:13:48","guid":{"rendered":"https:\/\/smilegate.ai\/?p=59805"},"modified":"2021-01-21T09:13:50","modified_gmt":"2021-01-21T00:13:50","slug":"horovod-raysgd","status":"publish","type":"post","link":"https:\/\/smilegate.ai\/cn\/2021\/01\/21\/horovod-raysgd\/","title":{"rendered":"\u5206\u5e03\u5f0f\u5b66\u4e60\u6846\u67b6\uff1aHorovod\u548cRaySGD"},"content":{"rendered":"<p>\u968f\u7740\u6df1\u5ea6\u5b66\u4e60\u6a21\u578b\u7684\u89c4\u6a21\u5448\u6307\u6570\u589e\u957f\uff0c\u4e0d\u518d\u96be\u4e8e\u901a\u8fc7\u4e00\u53f0\u673a\u5668\u5b9e\u73b0\u53ef\u7528\u7684\u5b66\u4e60\u65f6\u95f4\u3002 GPT-2\u662f\u4e00\u79cd\u8457\u540d\u7684\u4f1a\u8bdd\u6a21\u578b\uff0c\u5177\u6709\u7ea61.5B\u7684\u53c2\u6570\uff0c\u636e\u8bf4\u5df2\u4f7f\u7528\u4e86800\u4e07\u4e2a\u7f51\u9875\u8fdb\u884c\u57f9\u8bad\u3002 GPT-3\u5177\u6709175B\u53c2\u6570\uff0c\u662fGPT-2\u7684100\u500d\u4ee5\u4e0a\uff0c\u8981\u5728\u6b64\u7ea7\u522b\u8bad\u7ec3\u6a21\u578b\uff0c\u9700\u8981\u6784\u5efa\u5927\u578bGPU\u96c6\u7fa4\u3002<\/p>\n\n\n\n<p>\u8457\u540d\u7684\u5b66\u4e60\u6846\u67b6\uff08\u4f8b\u5982Tensorflow\u548cPyTorch\u7684\u6700\u65b0\u7248\u672c\uff09\u5305\u62ec\u5206\u5e03\u5f0f\u5b66\u4e60\u529f\u80fd\uff0c\u8be5\u529f\u80fd\u5229\u7528\u591a\u4e2aGPU\u673a\u5668\u8fdb\u884c\u57f9\u8bad\u3002\u4f46\u662f\uff0c\u914d\u7f6e\u5f88\u590d\u6742\uff0c\u4e3a\u4e86\u8fbe\u5230\u6b63\u5e38\u7684\u6548\u7387\uff0c\u60a8\u5fc5\u987b\u5728\u8bb8\u591a\u9886\u57df\u4e2d\u6267\u884c\u9ad8\u96be\u5ea6\u7684\u4efb\u52a1\uff0c\u4f8b\u5982\u7f51\u7edc\u914d\u7f6e\uff0c\u6743\u9650\u7ba1\u7406\u548c\u6570\u636e\u5171\u4eab\u3002\u56e0\u6b64\uff0c\u6b63\u5728\u51fa\u73b0\u4e00\u4e9b\u65e8\u5728\u8fdb\u884c\u6709\u6548\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u7684\u6846\u67b6\u3002<\/p>\n\n\n\n<p>Horovod\u662f\u4e00\u4e2a\u8457\u540d\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u6846\u67b6\uff0c\u5b83\u652f\u6301Keras\uff0cTensorflow\uff0cPyTorch\u548cMxNet\uff1a<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n<div class=\"vlp-link-container vlp-layout-basic wp-block-visual-link-preview-link\"><a href=\"https:\/\/github.com\/horovod\/horovod\" class=\"vlp-link\" title=\"\u970d\u7f57\u6c83\u5fb7\" rel=\"nofollow noopener\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/avatars2.githubusercontent.com\/u\/46361271?s=400&amp;v=4\" style=\"max-width: 150px; max-height: 150px\" alt=\"\"><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">\u970d\u7f57\u6c83\u5fb7<\/div><div class=\"vlp-block-1 vlp-link-summary\">\u9488\u5bf9TensorFlow\uff0cKeras\uff0cPyTorch\u548cApache MXNet\u7684\u5206\u5e03\u5f0f\u57f9\u8bad\u6846\u67b6\u3002 \u2013 Horovod \/ horovod<\/div><\/div><\/div>\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p>\u6839\u636e\u4f7f\u7528Horovod\u8fdb\u884c\u7684\u5b9e\u9a8c\uff0c\u5728Inception V3\u6216ResNet-101\u7684\u60c5\u51b5\u4e0b\uff0c\u4e0e\u5355\u4e2a\u8282\u70b9\u76f8\u6bd4\uff0c\u53ef\u4ee5\u83b7\u5f9790%\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u6548\u7387\uff0c\u5728VGG-16\u7684\u60c5\u51b5\u4e0b\uff0c\u53ef\u4ee5\u83b7\u5f9768%\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u6548\u7387\u3002\u6362\u53e5\u8bdd\u8bf4\uff0c\u4f8b\u5982\uff0c\u5982\u679c\u4f7f\u75284\u4e2a\u8282\u70b9\uff0c\u5219Inception V3\u7684\u5b66\u4e60\u6548\u7387\u662f3.6\u500d\uff0cVGG-16\u7684\u5b66\u4e60\u6548\u7387\u662f1.8\u500d\u3002<\/p>\n\n\n\n<p>RaySGD\u662f\u5728PyTorch\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u529f\u80fd\u4e4b\u4e0a\u5b9e\u73b0\u7684\u6846\u67b6\uff0c\u65e8\u5728\u6781\u5927\u5730\u63d0\u9ad8\u8bbe\u7f6e\u7684\u4fbf\u5229\u6027\u3002\u5c40\u9650\u5728PyTorch\u4e0a\u662f\u6709\u7f3a\u70b9\u7684\uff0c\u4f46\u662f\u4e0eHorovod\u76f8\u6bd4\uff0c\u5206\u5e03\u5f0f\u5b66\u4e60\u7684\u6548\u7387\u7565\u6709\u63d0\u9ad8\uff0c\u636e\u8bf4\u4e0e\u73b0\u6709PyTorch\u5185\u7f6e\u7684\u5206\u5e03\u5f0f\u5b66\u4e60\u529f\u80fd\u76f8\u6bd4\uff0c\u5927\u7ea6\u63d0\u9ad8\u4e8620%\u6216\u66f4\u591a\u3002\u7279\u522b\u662f\uff0cHorovod\u5728\u8bbe\u7f6e\u5916\u90e8\u5e93\uff08\u4f8b\u5982MPI\u6216NCCL\uff09\u65b9\u9762\u6709\u70b9\u590d\u6742\uff0c\u5e76\u4e14\u4e0e\u5fc5\u987b\u6839\u636e\u73af\u5883\u8fdb\u884c\u6784\u5efa\u548c\u4f7f\u7528\u76f8\u6bd4\uff0cRaySGD\u7684\u4f18\u52bf\u662f\u4ec5\u901a\u8fc7\u7b80\u5355\u7684\u5b89\u88c5\u548c\u914d\u7f6e\u5373\u53ef\u521b\u5efa\u53ef\u6269\u5c55\u7684\u5b66\u4e60\u4ee3\u7801\uff1a<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n<div class=\"vlp-link-container vlp-layout-basic wp-block-visual-link-preview-link\"><a href=\"https:\/\/medium.com\/distributed-computing-with-ray\/faster-and-cheaper-pytorch-with-raysgd-a5a44d4fd220\" class=\"vlp-link\" title=\"\u4f7f\u7528RaySGD\u66f4\u5feb\u66f4\u4fbf\u5b9c\u7684Pytorch\" rel=\"nofollow noopener\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/max\/1200\/0*UHI6GR-pOcb7Alxl\" style=\"max-width: 150px; max-height: 150px\" alt=\"\"><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">\u4f7f\u7528RaySGD\u66f4\u5feb\u66f4\u4fbf\u5b9c\u7684Pytorch<\/div><div class=\"vlp-block-1 vlp-link-summary\">\u5206\u5e03\u5f0f\u57f9\u8bad\u5f88\u70e6\u4eba\uff0c\u800c\u4e14\u8fd0\u884c\u8d77\u6765\u5f88\u6602\u8d35\u3002\u8fd9\u662f\u4e00\u4e2a\u4f7f\u5206\u5e03\u5f0fPytorch\u57f9\u8bad\u7b80\u5355\u800c\u4fbf\u5b9c\u7684\u5e93\u3002<\/div><\/div><\/div>\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p>Horovod\u548cRaySGD\u90fd\u662f\u514d\u8d39\u7684\uff0c\u5f00\u6e90\u7684\u5e76\u4e14\u6b63\u5728\u79ef\u6781\u6539\u8fdb\u9879\u76ee\u3002\u5f53\u7136\uff0c\u4f7f\u7528\u9884\u6784\u5efa\u7684GPU\u96c6\u7fa4\uff08\u4f8b\u5982AWS\uff09\u662f\u4e00\u79cd\u65b9\u6cd5\uff0c\u4f46\u662f\u51fa\u4e8e\u6784\u5efa\u81ea\u5df1\u7684GPU\u573a\u7684\u76ee\u7684\uff0c\u5f15\u5165\u548c\u4f7f\u7528\u8fd9\u6837\u7684\u6846\u67b6\u5c06\u662f\u4e00\u4e2a\u4e0d\u9519\u7684\u8d77\u70b9\u3002 <\/p>\n<div class=\"likebtn_container\" style=\"\"><!-- LikeBtn.com BEGIN --><span class=\"likebtn-wrapper\"  data-identifier=\"post_59805\"  data-site_id=\"5f255b28943ec9a60bed08fd\"  data-lang=\"ko\"  data-dislike_enabled=\"false\"  data-icon_dislike_show=\"false\"  data-tooltip_enabled=\"false\"  data-white_label=\"true\"  data-rich_snippet=\"true\"  data-popup_disabled=\"true\"  data-i18n_like=\"Like\"  data-style=\"\"  data-unlike_allowed=\"\"  data-show_copyright=\"\"  data-item_url=\"https:\/\/smilegate.ai\/cn\/2021\/01\/21\/horovod-raysgd\/\"  data-item_title=\"\ubd84\uc0b0 \ud559\uc2b5 \ud504\ub808\uc784\uc6cc\ud06c: Horovod\uc640 RaySGD\"  data-item_image=\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/01\/horovod-800x533-1.png\"  data-item_date=\"2021-01-21T09:13:48+09:00\"  data-engine=\"WordPress\"  data-plugin_v=\"2.6.55\"  data-prx=\"https:\/\/smilegate.ai\/wp-admin\/admin-ajax.php?action=likebtn_prx\"  data-event_handler=\"likebtn_eh\" ><\/span><!-- LikeBtn.com END --><\/div>","protected":false},"excerpt":{"rendered":"<p>\ub525\ub7ec\ub2dd \ubaa8\ub378\ub4e4\uc758 \ud06c\uae30\uac00 \uae30\ud558\uae09\uc218\uc801\uc73c\ub85c \uc99d\uac00\ud558\uba74\uc11c \ub354 \uc774\uc0c1 \ud558\ub098\uc758 \uba38\uc2e0\uc73c\ub85c\ub294 \ud65c\uc6a9 \uac00\ub2a5\ud55c \uc218\uc900\uc758 \ud559\uc2b5 \uc2dc\uac04\uc744 \ub2ec\uc131\ud558\uae30\uac00 \uc5b4\ub824\uc6cc\uc9c0\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc798 \uc54c\ub824\uc9c4 \ub300\ud654 \ubaa8\ub378\uc778 GPT-2\ub294 \uc57d 1.5B\uac1c\uc758 \ud30c\ub77c\ubbf8\ud130\ub97c \uac00\uc9c0\uace0 \uc788\uc73c\uba70 \ud559\uc2b5\uc744 \uc704\ud574 8\ubc31\ub9cc\uac1c\uc758 \uc6f9\ud398\uc774\uc9c0\ub97c \uc0ac\uc6a9\ud588\ub2e4\uace0 \ud569\ub2c8\ub2e4. GPT-3\ub294 GPT-2\uc5d0 \ube44\ud574 100\ubc30 \uc774\uc0c1\uc778 175B\uac1c\uc758 \ud30c\ub77c\ubbf8\ud130\ub97c \uac00\uc9c0\uace0 \uc788\uc73c\uba70 \uc774 \uc815\ub3c4 \uc218\uc900\uc758 \ubaa8\ub378\uc744 \ud559\uc2b5\ud558\ub824\uba74 \ub300\uaddc\ubaa8\uc758 GPU \ud074\ub7ec\uc2a4\ud130\ub97c \uad6c\ucd95\ud574\uc57c \ud569\ub2c8\ub2e4. \ucd5c\uadfc \ubc84\uc804\uc758 Tensorflow\ub098&hellip;<\/p>\n<div class=\"likebtn_container\" style=\"\"><!-- LikeBtn.com BEGIN --><span class=\"likebtn-wrapper\"  data-identifier=\"post_59805\"  data-site_id=\"5f255b28943ec9a60bed08fd\"  data-lang=\"ko\"  data-dislike_enabled=\"false\"  data-icon_dislike_show=\"false\"  data-tooltip_enabled=\"false\"  data-white_label=\"true\"  data-rich_snippet=\"true\"  data-popup_disabled=\"true\"  data-i18n_like=\"Like\"  data-style=\"\"  data-unlike_allowed=\"\"  data-show_copyright=\"\"  data-item_url=\"https:\/\/smilegate.ai\/cn\/2021\/01\/21\/horovod-raysgd\/\"  data-item_title=\"\ubd84\uc0b0 \ud559\uc2b5 \ud504\ub808\uc784\uc6cc\ud06c: Horovod\uc640 RaySGD\"  data-item_image=\"https:\/\/smilegate.ai\/wp-content\/uploads\/2021\/01\/horovod-800x533-1.png\"  data-item_date=\"2021-01-21T09:13:48+09:00\"  data-engine=\"WordPress\"  data-plugin_v=\"2.6.55\"  data-prx=\"https:\/\/smilegate.ai\/wp-admin\/admin-ajax.php?action=likebtn_prx\"  data-event_handler=\"likebtn_eh\" ><\/span><!-- LikeBtn.com END --><\/div>","protected":false},"author":1,"featured_media":59807,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[196],"tags":[430,431,432],"class_list":["post-59805","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-code","tag-horovod","tag-raysgd","tag-distributed-training","category-196","description-off"],"_links":{"self":[{"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/posts\/59805","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/comments?post=59805"}],"version-history":[{"count":1,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/posts\/59805\/revisions"}],"predecessor-version":[{"id":59806,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/posts\/59805\/revisions\/59806"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/media\/59807"}],"wp:attachment":[{"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/media?parent=59805"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/categories?post=59805"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/smilegate.ai\/cn\/wp-json\/wp\/v2\/tags?post=59805"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}